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ABSTRACT 



This study examines the importance of using effectiveness 
indicators for assessing institutional outcomes, and their congruence with 
actual or potential use in making institutional change and improvement, in 
selected northwest community colleges. The study was undertaken in response 
to the Northwest Association of Schools and Colleges’ transition to an 
outcomes-based accreditation model in 1991. Questionnaires were completed by 
148 primary administrators working at northwest community colleges that 
completed the regional accreditation process since 1990. The survey 
instrument measured whether 13 effectiveness indicators were important to the 
institutional change process, and whether such indicators have been used at 
the institution. In general, the study shows that there is a positive 
relationship in community colleges between the development of institutional 
effectiveness indicators and institutional change and improvement. 
Overwhelming support was given by administrators about the value of using 
effectiveness indicators to initiate change, with support ranging from 85.5% 
to 97.0% they should use effectiveness indicators to initiate institutional 
change. Indicators prioritized as most important by at least 90% of 
participants include: external indicators related to employer satisfaction 
and job placement levels, student and customer satisfaction, and student 
progression. All administrators who have used effectiveness indicators 
believe that external indicators and community development led to the most 
positive change. (Contains approximately 90 references.) (AS) 
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In 1990, the Northwest Association of Schools and Colleges moved to an 
accreditation model based on outcomes, their assessment and deliberative attempts at 
institutional change and improvement. No study existed in northwest community colleges 
which identified the systematic application of effectiveness indicators for institutional 
change and improvement. This study was conducted with primary administrators in 
northwest community colleges that have completed the regional accreditation process 
since 1990. It examined the perceived importance of using effectiveness indicators for 
assessing institutional outcomes, and their congruence with actual or potential use in 
making institutional change and improvements. 

This study shows there is a positive relationship in community colleges between 
the development of institutional effectiveness indicators and institutional change and 
improvement. All administrators believed they should be using effectiveness indicators but 
used them in varying degrees for planning, improving services, budgeting, accreditation. 
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program review and other decision making processes essential to the functioning of 
community colleges. Although the use varied by category and administrative subgroup, 
those administrators who had used effectiveness indicators believed their use led to 
positive institutional change. 

In order for this study to have an affect on northwest community colleges, several 
important conditions need to be addressed. The Northwest Association of Schools and 
Colleges must enforce Policy Statement 2.2, Policy on Educational Assessment, which 
requires the adoption of an institutional effectiveness model. Community colleges must 
embrace this model as a mechanism useful for the continuous improvement process. 
Faculty, staff, administrators and boards must be trained in the application and use of the 
institutional effectiveness model. Boards must establish outcomes or institutional goals 
based on their unique mission and must hold the community college president accountable 
for the achievement of institutional outcomes and the implementation of an assessment 
process. Assessment must be completed at all levels of the institution. Institutions must 
adopt an assessment model that reflects the institutions goals and objectives, and place its 
responsibility prominently within the organization. Finally, measures must be developed at 
each level or process by the persons closest to the process. 

Failure to enforce accreditation standards will lead to publicly enacted performance 
indicators aimed at achieving institutional effectiveness or accountability. 
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An Investigative Study on the Systematic Application 
of Effectiveness Indicators for Institutional Improvement 
in Northwest Community Colleges 

Chapter I: Introduction 
Background 

Education, at all levels, is under constant pressure from individuals representing 
public and private sectors, for change and improvement. Tax limitation legislation, federal 
and state legislative mandates, workforce training needs, accreditation requirements, 
quality improvement philosophies, and most important of all, the contemporary 
educational needs of students require institutions to change and improve. This context has 
motivated community colleges to identify and use effectiveness indicators as a strategy 
aimed at purposeful change and improvement in an effort to fulfill their unique mission. 

James Ratcliff described community colleges as “.. .those institutions that provide 
general and liberal education, career and vocational education, and adult and continuing 
education. . . . The term ‘community college’ is now used generically to refer to all 
colleges awarding no higher than a two-year degree” (Ratcliff in Baker III, 1994, p. 4). 

Community colleges within the system of all higher education in the United States 
fulfill a particular mission. This higher education system was described by the Carnegie 
Classification as “. . . all colleges and universities in the United States that are degree- 
granting and accredited by an agency recognized by the U.S. Secretary of Education” 
(AACU, 1996, p. 52). The classification of institutions is described hierarchically as 
Research Universities, Doctoral Universities, Masters or Comprehensive Colleges and 
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Universities, Baccalaureate or Liberal Arts Colleges and Associate of Arts Colleges which 
include community, junior and technical colleges (AASCU, 1996). 

Discussions about this unique position of community colleges in American 
education began as early as 1894 by Reverend Jim Carroll of Baylor University, and 
almost immediately repeated by William Rainey Harper of the University of Chicago and 
David Starr Jordan, President of Stanford University. The early founders of junior 
colleges supported them for different reasons but the general concept of fulfilling the 
freshman and sophomore years of a baccalaureate or equivalent education was the 
intended purpose (Ratcliff in Baker III, 1994). 

From these early years the evolution of community colleges in the United States 
has fulfilled a mission described primarily as focused on access. Ratcliff described the 
mission succinctly. “Community Colleges have stood for open admissions, geographic 
proximity, and relative financial affordability to the potential students of the community 
and region served. Within this structure of American higher education, the community 
college’s contribution has been increased accessibility and pragmatic curricular diversity 
geared to local and regional needs” (Ratcliff in Baker III, 1994, p. 4). 

The effectiveness of community colleges in meeting their mission may be 
experienced directly by students in classrooms or identified by “proxy” indicators or 
indicators of effectiveness. Proxy indicators might include occupational placement rates 
or rates of transfer to four year colleges and universities which by assumption indicate 
effectiveness in meeting this mission. In the absence of indicators, effectiveness is 
informally interpreted from individual opinions, media presentations, and anecdotal 
information. Opinions about effective or ineffective education will always be held by the 
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public and may be based on the best information available or upon readily available 
anecdotal information. Community colleges operating in this context have, out of 
necessity, been motivated to develop effectiveness indicators which provide the best 
information available for decision-making, program improvements and molding of public 
opinion. If community colleges are to improve, most community college leaders believe 
they must systematically develop objectives and strategies for program and process 
improvement (Ewell & Jones, 1994; AACC, 1994). 

The development and understanding of appropriate effectiveness indicators, 
accepted internally by members of the institutions and externally by the public is difficult. 
The use of effectiveness indicators for purposeful systematic institutional change and 
improvement is even more challenging since needed improvements compete with other 
politically important or educationally significant needs. 

Institutional Effectiveness as an Accreditation Requirement 

Accreditation is usually a process of evaluation performed by peers from the 
region. The Northwest Association of Schools and Colleges described accreditation as 

... a process of recognizing educational institutions for performance, 
integrity, and quality which entitles them to the confidence of the educational 
community and the public. In the United States this recognition is extended largely 
through non-governmental, voluntary institutional or professional associations 
which have responsibility for establishing criteria, visiting and evaluating 
institutions at their requests and approving those institutions and programs which 
meet their criteria. (NASC, 1994, p. 1) 

Two forms of accreditation exist in community colleges. The institutional 
accreditation is granted only by a regional accrediting commission or association and 
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specialized accreditation is granted by national organizations representing professions such 
as law, medicine, engineering, architecture, or auto mechanics. 

This study is concerned with institutional accreditation and the requirements for 
that accreditation. The Northwest Association of Schools and Colleges was originally 
formed in 1917 and has been listed by the “U.S. Secretary of Education as a nationally 
recognized accrediting agency for institutions offering programs of at least one academic 
year in length at the post-secondary level” (NASC, 1994, p. 4). 

Accreditation requirements are recognized as one of the three strongest influences 
on community colleges. The other two are legislative or public policy mandates and 
bureaucratic agency directives either from the federal or state level. 

The intrusiveness of federal agency directives was reflected by actions of the U.S. 
Department of Education and epitomized by their development of the State Postsecondary 
Review Entities (SPRE) later struck down by Congress. The SPRE was intended to 
directly or indirectly give more accreditation control to the U.S. Department of Education 
in concert with appropriate state agencies. What it did most was add weight to the 
already adopted outcome assessment standards of all regional accrediting associations and 
more directly involved federal agencies in this process. 

This federal attitude and action was the culmination of many previous commission 
reports and probably the result of their perception about the lack of perceived use of 
effectiveness indicators in many educational two and four year institutions of higher 
education. The regulation, although struck down by Congress in the 1995 session, 
epitomized the attitude of many state and federal agency staff and politicians. It was 
conceived in a hostile educational environment. It: (a) placed more control in the hands of 
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federal and state government over education; (b) reflected a lack of trust in public 
institutions; (c) reflected a lack of trust in self regulation; (d) reflected some scandals in 
public institutions; (e) reflected the high cost of education and need for access; (f) 
reflected the lack of graduating student skills; and (g) reflected the lack of repayment of 
student loans by students (Malik and Petersen, 1993, p. 15). 

Although the proposed change failed, “it is clear that the department’s career staff 
seek regulatory power over the non-governmental accrediting associations and intend to 
force accreditors in turn to regulate public and independent nonprofit institutions on behalf 
of the department” (Malik and Petersen, 1993, p. 16). 

Although these intrusive measures seem to be ongoing, regional accrediting 
associations did begin to formalize outcomes assessment requirements as early as 1985. 

The 1990 Annual Report of the Commission on Colleges of the Southern 
Association of Colleges and Schools advocated a shift in operational philosophy; 
...the shift from a set of standards which were essentially input driven to a set of 
criteria with institutional effectiveness as the integrating principle represented a 
major change, one that would soon be emulated by other accrediting entities. 
(Bryant, 1994, p. 18) 

In the Pacific Northwest, the Northwest Association of Schools and Colleges, 

Commission on Colleges, also shifted its philosophy and began to require outcomes 
assessment through the use of effectiveness indicators by the adoption of the Standard V, 
Policy #25, in 1990, which in 1996, became Standard Two - Educational Program And Its 
Effectiveness: Policy 2.2, Policy on Educational Assessment. 
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Institutional Effectiveness-A Requirement of the Northwest Association of Schools and 
Colleges. Commission on Colleges: Standard V. Policy #25 

Under the 1990 requirements of the Northwest Association of Schools and 

Colleges, Commission on Colleges, “Standard V - Educational Program and its 

Effectiveness” requires outcomes assessment. In Standard V: 

...the Commission emphasizes the necessity of a continuing process of academic 
planning, the carrying out of those plans, the assessment of outcomes, and the 
influencing of the planning process by the assessment activities. Assessment which 
does not result in conscious and deliberative attempts at improvement is rarely 
worth the effort. (NASC, 1994, p. 51) 



The policy statement requires institutions to develop an institutional effectiveness model 
or plan for institutional improvement even though the specificity of the model is left open 
to each institution. 

The commission, in 1996, in Policy Statement 2.2 Policy on Educational Assessment: 

...expects each institution and program to adopt an assessment plan 
responsive to its mission and its needs. In so doing, the commission urges the 
necessity of a continuing process of academic planning, the carrying out of those 
plans, the assessment of the outcomes, and the influencing of the planning process 
by the assessment activities. (NASC, 1996, p. 36) 



The commission continued to explain how accreditation has changed historically from a 
process which looked at resources, processes, and structures to one which emphasizes 
outcomes “to evaluate the effectiveness of that educational program in terms of the 
change it brings about in students, and to make improvements in the program dictated by 
the evaluative process” (NASC, 1994, p. 58). The Commission on Colleges went on to 
claim that “nearly every post secondary institution accredited by the Commission on 
Colleges engages in some type of outcomes assessment” (NASC, 1994, p. 5). 
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This study will examine the relationship of this requirement to actual attitudes and 
behaviors of selected administrators in northwest community colleges which have 
undergone accreditation since 1 990 when the NASC requirement began. The commission 
stated in 1 990, 

The ...policy is to stress outcome assessment as an essential part of the 
ongoing institutional self study and accreditation processes to underline the 
necessity for each institution to formulate a plan which provide for a series of 
outcome measures that are internally consistent and in accord with its mission and 
structure, and finally to provide some examples of a variety of successful schemes 
for assessing educational outcomes. (NASC, 1994, p. 58; NASC, 1996, p. 37) 

Accreditation Guidelines and Indicators of Effectiveness 

The 1 996 edition, as well as earlier versions of the Accreditation Handbook 
produced since 1990, by the Northwest Association of Schools and Colleges (NASC) has 
provided several guidelines which would assist colleges in developing a list of potential 
indicators to be used in assessing institutional effectiveness. NASC states: “There follows 
several outcomes measures which when used in appropriate combinations and informed by 
institutional mission, could yield an efficacious program of outcomes assessment. This list 
is intended to be illustrative and exemplary as opposed to prescriptive and exhaustive” 
(NASC, 1996, p. 37). 

The kinds of indicators suggested and used in conjunction with the unique mission 
of each community college are: “student information,” such as gender, age group, and 
ethnicity over time; “mid-program assessments,” such as pre and post writing samples in 
the completion of writing courses; “end of program assessments,” such as retention 
pattern of students over time; “program review and specialized accreditation,” such as 
nursing accreditation by National League of Nursing or a regular program evaluation 
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process by internal or external reviewers; “alumni satisfaction and loyalty,” such as 
customer satisfaction surveys about the focus and quality of their educational experience; 
“dropouts and non-completers,” such as attrition rates over time or attrition rates in 
relationship to student success strategies; “employment and/or employer satisfaction 
measures,” such as employer follow-up studies on the quality of preparation of trained 
workers or follow-up studies with graduates about the quality and satisfaction with their 
training and its applicability to their current employment (NASC, 1996, p. 37-38). 

The NASC as well as other regional accrediting associations have not tried to 
directly prescribe which indicators to use, however when examining the list of indicators 
used by Midlands Community Colleges, or the core indicators suggested by the American 
Association of Community Colleges, one can quickly recognize almost all of the 
suggestions made by the NASC, particularly when developed in conjunction with the 
mission goals and objectives required of each community college through accreditation, if 
not by normal practice. 



The Study 



Problem Statement. 

This study will examine the perceived importance of using effectiveness indicators 
for assessing institutional outcomes, and their congruence with actual or potential use in 
making institutional changes and improvements in northwest community colleges that have 
completed the regional accreditation process since 1990. 
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No study exists in northwest community colleges which identifies the systematic 
application of effectiveness indicators for institutional change and improvement. This is an 
important study because the previous accreditation requirement in “Standard V” and 
currently Policy 2.2 of the Northwest Association of Schools and Colleges, beginning as 

V 

early as 1990, imply a relationship between accreditation self studies based on outcomes 
assessment and purposeful change and improvement in institutions. Roueche, Johnson 
and Roueche reiterated a similar theme in Embracing the Tiger: The Effectiveness Debate 
and the Community College. 1997). Johnson succinctly stated the issue when he said, 
“what has been lacking in the rush of literature that has emerged over the past decade is a 
picture of how community colleges have embraced the institutional effectiveness 
movement and how and to what extent they are measuring their own performance” 
(Johnson in Roueche, Johnson, Roueche, 1997, p. 27). 

Hypothesis. There is a positive relationship in community colleges accredited by 
Northwest Association of Schools and Colleges, Commission on Colleges since 1990, 
between the development of institutional effectiveness indicators and institutional change 
and improvement. 

General Questions. 

Questions that will set the context and guide the literature review are: 

1 . What kind of purposely selected effectiveness indicators are suggested by the 
accreditation process and related literature, and can be used to base 
institutional effectiveness assessment? 
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2. What does the research literature have to say about the use of effectiveness 
indicators in assessment practices and for making institutional change? 

Research Questions. 

Questions that will guide this investigative quantitative study are: 

1 . Do community college administrators believe they should use effectiveness 
indicators for initiating institutional change? 

2. Since the last community college regional accreditation (1990 or later) have 
community college administrators used effectiveness indicators for initiating 
institutional change? 

3. Do community college administrators believe that the use of effectiveness 
indicators led to positive changes in their institutions? 

4. Is there a difference by community college administrative position in the 
perception about the perceived importance of using effectiveness indicators for 
initiating institutional change? 

5. Is there a difference by community college administrative position in the use of 
effectiveness indicators for initiating institutional change? 

6. Is there a difference by community college administrative position in the 
perception that the use of effectiveness indicators led to positive changes in the 
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Institutional Effectiveness: The Issue 

Institutional effectiveness became important as an issue in community college 
education partially in response to the Carnegie Commission Report (1984) on quality, or 
the lack of quality, in higher education in the early 1980s. The National Governors 
Association gave impetus to the assessment movement in the mid 1980s by . .its shared 
vision that assessment is the catalyst for improving quality. In addition, the governors 
regarded assessment as a broader process of defining more clearly the mission of each 
institution and encouraging the use of information for program improvement” (Alexander, 
Clinton and Kean in Erwin, 1991, p. 2). 

The necessity for improvement in the outcomes of education was raised to a 
conscious level in community colleges by Dale Parnell in The Neglected Majority . (1985), 
in order to meet the educational and training needs of neglected high schools and 
community college students. In his most recent book “Why Do I Have To Learn This ?” 
(1995), he reiterated the importance of contextual teaching and competency-based 
educational practices as a strategy for meeting workforce training needs of the 21 st 
century, as stated in the Secretary’s Commission on Achieving Necessary Skills, U.S. 
Department of Labor (1992). Dan Hull and Dale Parnell again reiterated this theme in 
Tech Prep Associate Degree. (1991), in an attempt to develop contextual and competency 
based learning in high schools and community colleges and an improved level of student 
competencies. Quality learning in the classroom, the essence of effectiveness, was 
emphasized as a critical concern by Thomas Angelo and Patricia Cross in Classroom 
Assessment Techniques: A Handbook for College Teachers . (1993). They emphasized 
that classroom assessment techniques must focus on student feedback in the teaching and 





learning process in order for effective teaching and learning process to occur. As Pat 
Cross stated, “...teaching without learning is just talking” (Angelo & Cross, 1993). 

Characteristics of Successful Assessment Programs 
Educators endeavor to assess programs because of accreditation mandates, 
legislative requirements for accountability, funding enhancements, and a desire to improve 
the quality of the teaching and learning process. 

Assessment, whether it is directed at classroom teaching and learning processes or 
generally at institutional effectiveness, is difficult. It requires administration, faculty, and 
staff to be knowledgeable about technical measurements, content knowledge and skills, 
strategies for organizational change and improvement and an understanding of most 
institutional processes (Erwin, 1991, p. XI). 

Assessment and institutional improvement, through the use of effectiveness 
indicators, involves both the process of developing commitment to the need for 
improvement through the use of assessment techniques and the application of the 
institutional effectiveness model for assessing outcomes. 

Assessment of effectiveness requires a consistent level of understanding about 
qualitative and quantitative research methods and the patience to develop an institutional 
understanding about the value of both kinds of research (Erwin, 1991, p. XII). 

Erwin, in 1991, outlined the “characteristics of successful assessment programs” 
which reappear restated in subsequent literature and models as essential ingredients. 

First, successful assessment programs involve many people in the 
assessment process. . . . 

Second, there are clear, assessable educational goals and objectives. . . . 
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Third, other constituent groups have been brought into the process. . . . 

Fourth, the data collected are meaningful, valuable and accurate. . . . 

Fifth, the data are analyzed and not just tallied for compliance purposes. 
Efforts are made to study what the data mean for the program-what is working 
well and what needs improving. Programs that merely comply have poor survey 
return rates, no expectations about students’ performances, and no concern about 
why students performed better or worse. 

Last, a system is established for distributing and implementing assessment 
results so that the results are available to the people who can use them. Moreover, 
administrators make sure that the results are used. The active assessment process 
is integrated into the planning and evaluation process of the institution and the 
assessment information and reports support other review efforts, such as 
accreditation studies, or annual program reviews. In contrast, in unsuccessful 
programs, assessment information is distributed or understood by no one except 
the report writer. (Erwin, 1991, p. 24-25) 



Institutional Effectiveness as a Cyclical Process 
Nichols described the institutional effectiveness paradigm in his 1989 book. 
Institutional Effectiveness And Outcomes Assessment Implementation on Campus: A 
Practitioner’s Handbook, and again in 1991, in A Practitioner’s Handbook For 
Institutional Effectiveness And Student Outcomes Assessment Implementation . He 
described the elements of the cyclical process as containing: (a) Expanded mission 
statement; (b) identification of institutional and service objectives; (c) assessment of those 
objectives and how they are being accomplished; (d) and the revision, refinement and 
modification of those objectives based on assessment results. (Nichols, 1991). 

The following cyclical process is a visual interpretation of this institutional 
effectiveness paradigm as described and used by a number of institutional effectiveness 
models. 
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Figure 1 

Cyclical Process of The Institutional 
Effectiveness Model 
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Institutional Effectiveness as a Model 
Models of institutional effectiveness appear to be focused on the general 
philosophical principles which utilize much of Edward Deming’s philosophy of continuous 
improvement as an ongoing premise. Deming, often noted for being the father of Total 
Quality Management (TQM), based much of his drive for improvement as a focus on 
customers needs, assessment and continual improvement, and an understanding of 
systems. All of these are active ingredients in the process of doing assessment about 
effectiveness and the use of that information for institutional change and improvement 
(Deming, 1993). 
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Reasons for using effectiveness indicators at the state or local level can vary 
significantly. Purposes include compliance, persuading, monitoring, improving and 
judging. From these purposes various indicators are used to identify success. 

Effectiveness indicators may be categorized into six general types. First, “inputs” 
such as equitable access to institutions measured by enrollment. Second, “process 
indicators” such as climate studies and specialized accreditation studies. Third, “outcomes 
indicators” are currently the most common indicators used, include job placement rates, 
transfer and graduation rates. Fourth, “contextual indicators” are used and include 
funding level per FTE or facilities. Fifth, “return on investment” indicators are used to 
identify the net benefit of participating in a community college education based on the cost 
of participation. Sixth, “customer satisfaction” indicators are used and include student and 
employer, or transfer institution satisfaction (Ewell & Jones, 1994, p. 10-12). 
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Figure 2 

Six Types of Effectiveness Indicators: 
Three Levels of Focus 
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Six Types of Effectiveness Indicators 
Three Levels of Focus 




Most institutional models use a variety of indicators or categories of indicators in 
their models of institutional effectiveness, and although outcomes indicators are most 
common, inputs, process, outcomes, contextual, return on investment, and customer 
satisfaction indicators are all employed (Ewell & Jones, 1994). 

Definition of Terms 

Community colleges are defined here as those two year public institutions or 
baccalaureate institutions accredited to offer associate degrees. Proprietary institutions 
that may also offer associate degrees are not included in the study. 
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The language of assessment as with other segments of education carries with it a 
number of useful but specifically defined terms. In this study about institutional 
effectiveness the following definitions are used for clarification and understanding: 

1 . Institutional Effectiveness (or) Effective Community Colleges are defined as engaging 

...in successful transactions with important groups (public). They are able to 
compare the results achieved (performance) with needs and expectations. They 
provide information about performance in ways that build understanding 
(perception) of their mission and purpose. Effectiveness suggests that a college 
has a discernible mission, is producing outcomes that meet constituency needs, and 
can conclusively document the outcomes it is producing as a reflection of its 
mission. (AACC, 1994) 

2. Assessment is defined as “. . .the process of descriptively evaluating the product of the 
institution in an objective manner” (McLeod & Atwell, 1992, p. 31). 

3. Baseline “...is simply a beginning measure of a given indicator” (McLeod & Atwell 
1992, p. 31). 

4. Benchmark in comparison to baseline is “...an absolute standard as opposed to a 
normative one” (McLeod & Atwell, 1992, p. 32). 

5. Indicator is “...a concrete piece of information about a condition or result of public 
action that is regularly produced, publicly reported, and systematically used for planning, 
monitoring, or resource allocation at the state or system level” (Ewell & Jones, 1994, 

p. 7). Indicators for the purpose of this study will be confined to local institutional use. 

6. Core indicator “...is a measure that describes a critical, widely recognized outcome of 
mission, one that is clearly responsive to key constituent groups and is regularly 
produced” (AACC, 1994, p. 9). 

7. Change is defined as comparing data over time. ‘Thus, change in indicators, or lack of 
change, is important for reaching conclusions about the effectiveness of an institution. 
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Key to this is that the same kind of data must be kept for comparison over extended 
periods of time” (McLeod & Atwell, 1992, p. 32). 

8. Family measure is defined as measurements representing “...a complex phenomenon 
with multiple measures, ‘...in that they are each separate entities, but they are related to 
one another. Together, they represent the entire situation. A family of measures clearly 
provides more, and more discrete, information about the total unit performance than a 
single measure, and this is more compatible with decision making in a context where 
trade-offs are common’ (Brinkeroff & Dressier, 1990, p. 37)” (McLeod & Atwell, 1992, 
p. 33). 

9. Mission is “...the second descriptive level of productivity for an institution. Whereas 
the initial statement of purpose is the first and is normally defined externally, usually in 
law, mission is usually internally defined” (McLeod & Atwell, 1992, p. 35). 

10. Objective is “...a written, active and operational subdivision (sometimes referred to as 
a specific objective) of a production goal. It must be well defined and, above all, 
measurable in its accomplishment. An objective should be specific respective to what is to 
be done, who is to do it, when is it to be completed, how it is to be evaluated and by what 
measure” (McLeod & Atwell, 1992, p. 35). 

11. Outcome is an “...end product. The productivity of an institution. Outcomes should 
not be confused with indicators. Outcomes are the products to be accomplished by 
specific objectives; indicators are the data which prove that the objectives have been 
accomplished” (McLeod & Atwell, 1992, p. 35). 

12. Two year colleges or community colleges include two year colleges granting associate 
degrees, defined here as community colleges who have undergone a full self study 
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required foraccreditation by the Northwest Association of Schools and Colleges, 
Commission on Colleges since 1990 when Standard V: Policy #25 was adopted in its 
current form. 



Research Design 

This quantitative study will be based on information collected in a self administered 
questionnaire using a cross-sectional study of a selected sample of community college 
administrators in northwest community colleges with the intent of generalizing the 
information for that population (Cresswell, 1994, p. 11). 

This “ . . quantitative study, consistent with the quantitative paradigm, is an inquiry 
into a social or human problem, based on testing a theory composed of variables, 
measured with numbers, and analyzed with statistical procedures, in order to determine 
whether the predictive generalizations of the theory hold true” Cresswell, 1994, p. 2). 

This is an investigative study in a descriptive style using simple descriptive and 
interactive statistics of relevant data to demonstrate perceived usefulness of institutional 
effectiveness indicators by administrators for making purposeful institutional change and 
improvement. 



Delimitation 

The study will limit its survey to CEOs or community college presidents and 
primary operational administrators in each institution including primary instructional 
administrators, primary student services administrators and primary business affairs 
administrators. The assumption is that accreditation as a requirement has mandated the 
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use of assessment information as a means to initiate institutional change and improvement. 
The accreditation requirement should influence the subsequent beliefs and actions of 
CEOs and primary operational administrators in each institution. These administrators 
with operational and budget responsibilities should consequently have an influence over 
the identification and use of assessment information for initiating change. The use of 
effectiveness indicators as one of the current models using assessment has been identified 
both in the literature and by accreditation associations as an appropriate model for 
addressing change and improvement. 

Boards of Trustees, although primarily responsible for the policy direction of 
institutions, were not surveyed based on the premise that the application and use of 
assessment information to initiate change had already been established as a policy by the 
accrediting association and therefore the policy direction was established. The application 
of the policy was subsequently up to the administration of each community college. 

The study was limited to CEOs and primary operational administrators based on 
the assumption that in the operational context of an organization, involving the planning, 
budgeting and evaluation processes, administrators who possess the budgeting 
responsibility for operational areas have a significant influence on what functions are 
supported or not supported and subsequently whether goals can be achieved. “Budgeting 
is a practical bridge between planning and evaluation. . . .Budgets represent the financial 
crystallization of an organizations intentions. It is through budgeting that a school or 
school district can decide how to allocate resources so as to achieve organizational goals” 
(Guthrie, Garms and Pierce, 1988, p. 216). 
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Therefore, one limitation of the study is that although the study was delimited to 
designated administrators, there is an obvious understanding that boards could influence 
the application and use of assessment information in positive and negative ways. The fact 
still remains that the direction of the policy was defacto established by the accreditation 
association. 



The Study 

A survey was conducted with purposely selected administrators at each college 
including the president, the primary instructional administrator, the primary student 
services administrator, and the primary business affairs administrator in northwest 
community colleges who have completed an outcomes based accreditation self study since 
1990 when the accreditation requirement began. 

The results were used to compare and contrast the perceptions, the use of 
institutional effectiveness indicators in community colleges and the application of those 
results for initiating purposeful institutional change and improvement. 

Data Analysis 

Simple descriptive and inferential statistics were used to analyze the perceived 
importance of institutional effectiveness indicators, their use in community colleges and 
whether they are perceived to have use for making purposeful institutional change and 
improvement. 





Significance of the Study 

In 1990, the Northwest Association of Schools and Colleges, Commission on 
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Colleges adopted a new standard which moved from an accreditation model based 
primarily on examining the input of resources into the educational process, to one based 
on outcomes, their assessment and deliberative attempts at institutional change and 
improvement. 

In Standard V, now Policy 2.2, the commission states: 

...the necessity of a continuing process of academic planning, the carrying 
out of those plans, the assessment of outcomes, and the influencing of the planning 
process by the assessment activities. Assessment which does not result in 
conscious and deliberative attempts at improvement is rarely worth the effort. 
(NASC, 1994, p. 51) 

Therefore, it is essential that administrators in community colleges, accredited by the 
Northwest Association of Schools and Colleges, have an understanding of institutional 
effectiveness models used for the purpose of initiating change, and have the belief that 
application of these models can produce institutional change or improvement. 

Limitations of the Study 

The study is limited by the fact that it is a cross-sectional descriptive survey of 
administrators in northwest community colleges whose institutions have gone through 
accreditation since 1990 and therefore any generalizations beyond this sample would be 
limited. The study is limited by the perceptions of those administrators in the sample and 
may not be reflective of all full and part time administrators in all northwest community 
colleges. The study is limited by the normal limitations of survey design and research 
related to this design. 
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Organization of the Study 

Chapter one has presented a review of the background on institutional 
effectiveness, accreditation requirements, a statement of the problem to be studied, 
hypothesis, research questions, purpose of the study, definition of terms, significance, 
delimitations and limitations of the study. 

Chapter two reviews appropriate literature pertaining to institutional effectiveness 
as related to the use of effectiveness indicators and general questions. 

Chapter three outlines the methods and procedures used in the research design. 

Chapter four presents an analysis of the research data. 

Chapter five presents study conclusions and recommendations of the study. 




tr>r 



24 



Chapter II: Review of Literature 

The foundation for assessment and institutional effectiveness in higher education 
began before the twentieth century and can be traced to the changes brought about by the 
Industrial Revolution with its significant impact on our social, cultural and political 
foundations. 

The following sections review the development of institutional effectiveness: first, 
as a construct derived from organizational effectiveness; second, as a model emanating 
from program evaluation; third, the forms of evaluation and assessment; and fourth, the 
application of institutional effectiveness models and the use of effectiveness indicators as 
required by accreditation and used for institutional change and improvement. 

Development of Institutional Effectiveness: Organizational Effectiveness 

Organizational Effectiveness is the first construct underlying the theory of 
institutional effectiveness. Early theorists in the development of organization effectiveness 
theory defined effectiveness in relation to other constructs or the social, cultural, political 
and economic thoughts of that period. 

The classical theorists of the time emanating from the "classical" school of 
organizational theorists, including Barnard, Taylor, Weber, Fayol, Gulick and Urwick, 
defined effectiveness of organizations in terms of "efficiency" (Spray, 1976, p.l). Early 
studies were based on these theoretical models. When joined with other disciplines of the 
sciences and social sciences, and applied to organizations, the results were a diversity of 
characteristics reflected by those theories and applied to organizational performance by 
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evaluations and measurements of the period (Spray, 1976, p.l). Many models developed 
after World War II and reflected the advances within many disciplines which implied their 
theoretical development. 

In recent years the case study tradition, initiated by sociologists such as Blau 
(1955), Gouldner (1954), Selznick (1949), and Whyte (1948) have been 
supplemented by a variety of approaches. These include the decision-making 
model of March and Simon (1958). The criterion approach of Georgopauulos and 
Tannenbaum (1971), the contingency approach of Lawrence and Lorsch (1967), 
the structural approach of the Aston group (Hinings and Lee, 1971; Inkson, Pugh, 
and Hickson, 1970; Pheysey, Payne and Pugh, 1971; Pugh and Pheysey, 1973), 
and various systems models (Evan, 1972; Katz and Kahn, 1966; Likert, 1961; 
Taylor and Bowers, 1972). In addition, a host of programmatic approaches, such 
as operations research, cost/benefit analysis, management by objectives, and 
various organizational development models, have recently been added to the 
literature on organizational effectiveness. (Spray, 1976, p. 1-2) 

Dubin raised the issue of a "fundamental dilemma" when assessing organizational 

effectiveness depending on whether one analyzes the organization from inside or outside. 

He considers the two views, “internal” perspective and “external” perspective. Internally 

he believes the organization asks "...whether the resources invested are utilized efficiently. 

Return on investment is typical to measure this type of organizational effectiveness" 

(Dubin in Spray, 1976, p. 7). Viewed externally, "a radically different view of 

organizational effectiveness consider the output of the organization, which is evaluated for 

its contribution to the larger society. Organizational effectiveness from this viewpoint is 

more likely to be measured by some form of cost-benefit analysis" (Dubin in Spray, 1976, 



The meshing of these two somewhat conflicting perspectives is present in the 
eclectic approaches of today’s models. Institutions today are interested in internal 
improvements which may have a relationship to efficiency and outcomes which are usually 
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the interests of legislators, taxpayers and even accrediting bodies. These external groups, 
as he indicated, seem to be most interested with outcomes of institutions (Hudgins, 1998; 
SBCTC, 1997; Borden & Bottrill in Borden and Banta, 1994). 

Organizational theory and organizational effectiveness models developed with 
corollary assessment indicators from the beginning of the twentieth century to date. All 
aspects of the theory appear to have some influence on the institutional effectiveness 
models currently promoted by public agencies as well as colleges and universities. 

One of the major influences on our current models is systems theory of 
organizations. 

From the systems theory-perspective, the organization is a social system which, in 
its interaction with its environment, activates at least four systemic processes: 

1 . Inputs (I) of various types of resources; 

2. Transformations (T) of resources with the aid of social and/or technical 
mechanisms; 

3. Outputs (O) which are transmitted to other systems, and 

4. Feedback effects (F) whether negative or positive. 

To appraise the effectiveness of an organization with the aid of systems theory 
. . . one must measure its performance with respect to all four systematic processes 
as well as their interrelationships. (Evan in Spray, 1976, p. 19) 

With the use of systems theory one can better understand the theoretical basis of today’s 

models which, although eclectic in nature, are based significantly on this theoretical 

construct of organizations hence educational institutions. 

Evan states that, 

. . .From the perspective of systems theory of organization, the four systemic 
processes outlined above suggest a new way of conceptualizing as well as 
operationalizing organizational effectiveness. A multidimensional concept, 
organizational effectiveness may be defined as the capacity of an organization to 
cope with all four systemic processes relative to its goal seeking behavior - 
however explicit or implicit this may be. (Evan in Spray, 1976, p. 20-21) 
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Two current models of organizational effectiveness and subsequent variations are 
based on different paradigms and different assumptions. The goal centered approach is 
dependent on quantitative measures in most cases and the natural systems approach is 
dependent on qualitative measures of performance and effectiveness. 

The Core Indicators of Effectiveness model developed by AACC, which is the 
primary foundation for the survey used in this study, more closely follows a goal centered 
approach and quantitative paradigm. Eclectic approaches using both quantitative and 
qualitative measures are becoming more popular, particularly at the classroom level of 
analysis (Angelo and Cross, 1993). 

The essence of an institutional decision making model is based on the following 
assumptions of a goal centered perspective: 

The goal centered view makes a reasonably explicit assumption that the 
organization is in the hands of a rational set of decision makers who have in mind a 
set of goals which they wish to pursue. Further, these goals are few enough in 
number to be manageable and can be defined well enough to be understood. Given 
that goals can be thus identified, it should be possible to plan the best management 
strategies for attaining them. Within this orientation, the way to assess 
organizational effectiveness would be to develop criterion measures to assess how 
well the goals are being achieved. (Campbell in Spray, 1976, p. 32) 



This goals-centered model based on overtones of systems theory of organizations 
seems to be the current foundation of models being developed for analyzing institutional 
effectiveness and particularly the approach suggested by the accrediting associations. 
(NASC, 1996). The methods used by institutions in analyzing their effectiveness does 
vary in practice. Both quantitative and qualitative methods are used in the assessment of 
effectiveness, although the construct or theory seems to be mostly goal centered. 
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Other models also exist for analyzing organizational effectiveness but are generally 
based on either a goal centered or natural systems construct. These models include: 

“Cost/ Benefit Analysis,” “Management by Objectives,” “Organizational Development 
Model,” “Likert ISR Model,” and the “Industrial/Organizational Psychology Criterion 
Model” (Campbell in Spray, 1976, p. 32-35). 

Development of Institutional Effectiveness: Program Evaluation 
Program Evaluation: 1800- 1900 

The foundations of program evaluation and consequently assessment, partially 
originated in the United States but more importantly in England. It began not in collegiate 
education but in primary education. "The Royal Commission of Inquiry into the Primary 
Education in Ireland under the Earl of Powis..." was one of the first official commissions 
set up to perform evaluation (Madaus, Stufflebeam & Scriven in Stark & Thomas, 1 994, 
p. 24). 

Other early forms of evaluation starting in Great Britain during the mid 19th 
century included associations which formed societies to evaluate the social problems of 
the day such as sanitary conditions of the working poor. Formal bureaucracies were 
developed to review the programs through enquiry committees (Madaus, Stufflebam & 
Scriven in Stark & Thomas, 1994, p. 24). 

In the United States, official evaluation of schools performance began as early as 
1845, through the use of student exams as a method of program evaluation. Horance 
Mann, among others, began to collect the test scores as an early method of evaluating the 
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effectiveness of programs and headmasters. Joseph Rice is credited with conducting the 
first long term educational program evaluation from 1887 to 1898, on the methods used in 
spelling instruction. Formal accreditation, somewhat as we know it today, began as early 
as the late 1800s when the North Central Association of Colleges and Secondary Schools 
was established (Madaus, Stufflebeam & Scriven in Stark & Thomas, 1994, p. 24-25). 
Four regions developed accrediting associations during these years but all were well 
established by the 1930s. The focus of this study, the Northwest Association of Schools 
and Colleges, was established in 1917 has been recognized by the U.S. Secretary of 
Education since 1952 as a nationally recognized accrediting agency (NASC, 1996). 



Program Evaluation: 1900- 1930 

During the early stages of the twentieth century “...scientific management became 
a powerful force in administrative theory as well as in industrial circles’ [Biddle & Ellena, 
1964; Callahan, 1962; Creanin, 1962.]. The emphasis of this movement was on 
systemization; standardization; and most importantly, efficiency” (Madaus, Stufflebeam & 
Scriven in Stark & Thomas, 1994, p. 25). Some of the more familiar subjects of 
assessment began at this time including studies on expenditures, dropout rates and the 
implementation of standardized tests on arithmetic, spelling and English in order to 
determine quality. During this period early offices of evaluation and measurement began 
to appear with formalized purposes. Developing from this were early normed formalized 
tests which enabled comparisons to be made between programs and classes around precise 
objectives. Many of these early forms were used to promote causes of those paying for 
expertise or trying to maintain the status quo. Although many evaluators were trying to 
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take the high road, most evaluations of this early period were localized and addressed 
local questions and not until later years did the generalization of the evaluations get 
addressed (Madaus, Stufflebeam & Scriven in Staark & Thomas, 1994, p. 25-26). 

Program Evaluation: 1931 - 1945 

The third significant period in program evaluation is 193 1 - 1945, known as the 
age of Ralph W. Tyler. Tyler's famous "Eight Year Study" evaluated the performance of 
intended versus actual outcomes of students attending traditional high schools and 
compared them to progressive schools based on their performance at cooperating colleges 
and universities. This "...approach calls for the measurement of behaviorally defined 
objectives, it concentrates on learning outcomes instead of organizational and teaching 
inputs ..." (Madaus, Stufflebeam & Scriven in Stark & Thomas, 1994, p. 27). This 
became the foundation for testing and evaluation for at least 25 years and is very close to 
the current outcomes based models suggested by institutional effectiveness literature and 
accreditation requirements since 1985 (NASC, 1996). 

Program Evaluation: 1946 - 1957 

The fourth significant period marks an expansion of all education, personnel and 
facilities including experimental and community colleges (Madaus, Stufflebeam & Scriven 
in Stark & Thomas, 1994, p. 27). During this period of growth, there was little concern 
about accountability in education. Although this period did not complete many program 
evaluations for accountability, it did expand technical standardized testing and standards 
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for evaluations. During this period the Educational Testing Service developed as well as 
other professional association standards around testing and evaluation. 

During this same period, based on much of Tyler’s work, was the development of 
"taxonomies of program objectives" and programs to help faculty and staff create and 
evaluate specific program objectives (Madaus, Stufflebeam & Scriven in Stark & Thomas, 
1994, p. 28). This was, however, still a localized and non-publicly supported venture. 

Program Evaluation: 1958 - 1972 

The fifth period, 1958-1972, grew from a quiet and subdued age into one calling 
for large scale projects and evaluations dependent on taxpayer support. The reaction to 
the scientific advantage demonstrated by the USSR with the launching of their first 
satellite called Sputnik, fueled the race for space and the launching of the National 
Defense Education Act of 1958 (Madaus, Stufflebeam & Scriven in Stark & Thomas, 

1994, p. 29; Gutek, 1986). 

New programs supported by federal funds included counseling and guidance as 
well as program evaluation. Program evaluations previously developed and refined were 
still used. Tyler's behavioral objectives, evaluations and standardized tests were all used as 
well as professional judgment and field experiments. Although all forms of evaluation 
were used, little seemed to be known about program effectiveness. Cronbach, in 1963, 
criticized evaluators for post hoc evaluations and asked researchers to examine and report 
findings which could be used to help guide curriculum and program changes (Madaus, 
Stufflebeam & Scriven in Stark & Thomas, 1994, p. 29). The “War On Poverty,” 
President Johnson’s new social program included educational programs and required 
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specific program evaluation to determine their effectiveness. The Elementary and 
Secondary Education Act of 1964, Title I required program evaluation to be used in 
practice (Gutek, 1 986). 

The standardized test approach could not be used successfully because it could not 
measure outcomes directly or the different objectives from area to area encouraged by the 
Tylerian model. 

New theories and methods had to be developed to evaluate the outcomes of 
education. New methods included a reformed Tylerian model of criterion-referenced 
testing, called systems-analysis. Others, such as Scriven, Stufflebeam, and Stake 
"...introduced new models of evaluation that departed radically from prior approaches. 
These conceptualizations recognized the need to evaluate goals, look at inputs, examine 
implementation and delivery services as well as measure intended and unintended 
outcomes of the program" (Madaus, Stufflebeam & Scriven in Stark & Thomas, 1994, p. 

3 1). Because many important studies found negative results in many of the popular 
educational and public programs of the day a separation between evaluation and research 
and testing began to occur. 

Program Evaluation: 1973 - 1980 

The sixth period developed during this time of separation and search by program 
evaluators for their own identity and professionalism. The period was filled with 
"...confusion, anxiety, and animosity. Evaluation had little stature and no political clout" 
(Madaus, Stufflebeam & Scriven in Stark & Thomas, 1994, p. 3 1). The period grew in 
professionalism with the advent of many new journals which "...proved to be excellent 
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vehicles for recording, and disseminating information about the various facets of program 
evaluation" (Madaus, Stufflebeam & Scriven in Stark & Thomas, 1994, p. 31). 

Colleges and universities began to offer courses to complete programs in the 
development of the profession. Much of the professional discussion during this period 
hinged on appropriateness of positivistic/quantitative versus phenomenological/qualitative 
approaches to evaluation and their related arguments (Madaus, Stufflebeam & Scriven in 
Stark & Thomas, 1994, p. 32). 

Program Evaluation: 1981 - 1995. A call for Education Reform through Evaluation and 
Assessment 

The seventh period marks the development of evaluation and assessment models in 
response to the call for education reform in the 1980s. Several major reports of the mid 
1980s called for educational reform. These included. To Strengthen Quality in Higher 
Education: Summary Recommendations of the National Commission on Higher Education 
Issues (1982), and A Nation at Risk: The Imperative of Educational Reform (Bennett, 
1983). These studies called for excellence in education and references to assessment as a 
mechanism for improvement (Nichols, 1991). Access to Quality Undergraduate 
Education by the Southern Regional Education Board, 1985, identified the linkage 
between basic skills deficiencies and the need for remediation of college freshman. 
Involvement in Learning: Realizing the Potential of American Higher Education. Final 
Report of the Study Group on the Conditions of Excellence in American Higher Education 
by the National Institute of Education in 1984, proposed “ . . .that individual student 
learning can be significantly enhanced through frequent communication about 
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performance... [and that]... institutions can also 'learn' through information about results 
and can make continuous improvements in response” (Ewell in Stark & Thomas, 1994, 
p.41), (NIE, 1984). 

Two additional reports. Integrity in the College Curriculum by the Association of 
American Colleges in 1985 and To Reclaim a Legacy: A Report on the Humanities in 
Higher Education by Bennett in 1984, “...are more traditionally focused in curricular 
content and structure. Here the curricular connection to assess lies largely in the felt need 
for intensive, integrative demonstrations of student knowledge and capacities (similar in 
structure to the comprehensive examinations typical of earlier undergraduate liberal arts 
curricula) to complete and certify the process of undergraduate instruction" (Ewell in 
Stark & Thomas, 1994, p. 41; Nichols, 1991; Bennett, 1984). 

In 1985 Secretary of Education William Bennett indicated that “colleges should 
state their goals, measure their success in meeting those goals, and make the results 
available to everyone. . . . If institutions don’t assess their own performance, others - 
either state or commercial outfits - will most likely do it for them” (Bennett, 1985, p. 25; 
Bennett in Nichols, 1991). In 1986, the Council on Postsecondary Accreditation (COP A) 
issued Educational Quality and Accreditation: A Call for Diversity Continuity and 
Innovation which asked institutions to 

• sharpen statements of mission and objectives to identify intended educational 
outcomes. 

• Develop additional effective means of assessing learning outcomes and results. 

• Use the self evaluation and peer review process of accreditation as an integral 
part of ongoing planning and institutional or programmatic change. (COP A, 
1986, p. 12) 
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It then asked accrediting bodies to evaluate these standards and achievements during the 
peer review process. 

The federal government in its recognition of the Council on Postsecondary 
Accreditation (COP A) in 1987, marked a fundamental change in accreditation policy and 
adopted an institutional effectiveness or outcomes assessment model for accreditation 
(Nichols, 1991). 

The federal change in accreditation policy was a focus on educational effectiveness 

and outlines that an accrediting agency insure that institutions maintain clearly specified 

educational objectives consistent with its mission through measuring its outcomes such as: 

. . satisfaction with certificate and degree requirements by all students. . . demonstrated 

educational achievement as assessed and documented through appropriate measures; 

. . . determining that institutions or programs document the educational achievements of 

their students; [and] . . . determining the extent to which institutions or programs 

systematically apply the information through the measures described. . .” (Federal Register 

CFR Part 602 in Nichols, 1991, p. 5). In 1992, COPA’s task force on Institutional 

Effectiveness reiterated these points in several additional papers (Ewell, 1992, p. 9). 

In addition to these commissions’ reports and special studies, state government 

became active in the call for education reform based on 

. . .the need to understand the performance of higher education in relation to the 
expenditures began to call for more and more performance criteria as a way of 
understanding and even supporting higher education. (Ewell in Stark & Thomas, 
1994, p. 42) 

In 1986 two major reports in this external environment were released. 
Transforming The State Role in Undergraduate Education: Time for a Different View 
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(Education Commission of the States, 1986) and Time for Results (National Govenors' 
Association, 1986). Consistent with prior academic improvement reports, these 
documents highlighted the use of assessment as a tool for reform. “...Both maintained that 
information on college student performances should be publicly available and comparable 
across institutions, should be used to inform policy and resource allocation decisions at the 
state level and should be appropriate to inform consumer choice on the part of students 
and their parents in the decision of which college to attend" (Ewell in Stark & Thomas, 
1994, p. 42; Nichols, 1991; National Governors’ Association, 1986, p. 161). 

In 1988 the AACC Commission on the Future of Community Colleges 
issued the landmark report Building Communities: A Vision for a New Century, 
With statements such as ‘The community college should be the nation’s premier 
teaching institution. Quality instruction should be the hallmark of the movement’ 
(p. 25), the Building Communities report helped usher in more than another 
document in recent community college literature, the emphasis on teaching and 
learning. (O’Banion, 1994, p. 22) 

In conjunction with the goal of being the “nation’s premier teaching institution” 
Building Communities asked the related assessment question, “What can be done to 
improve student outcomes and the effectiveness of the institution?” In response to this 
question AACC recommended several goals for assessment: 

[First,] . . . that classroom evaluation be the central assessment activity of 
the community college; [second,] . . .that each community college develop a 
campus- wide assessment of institutional effectiveness. Such a program should 
include a periodic reexamination of mission and goals, specific programs, 
individual student outcomes, retention rates and the performance of graduates; 
[third,] . . .those outcomes should be clearly related to the mission of the college 
and to an informed understanding of the educational needs and goals of the 
college’s student population. (AACC, 1988, p. 47-48) 



By 1990, based on accreditation requirements from the federal level and public 
policies and pressure, all regional accrediting associations including the Northwest 
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Association of Schools and Colleges had adopted some form of outcomes based 
accreditation or institutional effectiveness models following the initiative set by the 
Southern Association of Schools and Colleges as early as 1985 (Nichols, 1991; NASC, 
1994). 

The result of this intensive external involvement by state legislators and state 
educational agencies mandated assessment testing in a variety of formats within higher 
education. The involvement by state legislatures ranged from unfunded mandates to fully 
funded assessment programs. This context set the stage for the adoption of outcomes 
assessment by regional accrediting associations in the late 1980s and early 1990s as 
exemplified by Standard V, Policy Statement #25, now Standard 2, Policy 2.2 of the 
Northwest 'Association of Schools and Colleges, Commission on Colleges. Standard V, in 
essence requires outcomes assessment evaluations in self studies as a mandate of the 
accreditation process. 

This outcomes based assessment or institutional effectiveness model closely 
follows the “institutional effectiveness paradigm” developed by Nichols in the 1980s which 
called for: 

1 . A sharpened statement of institutional mission and objectives. 

2. Identification of intended departmental/programmatic outcomes or results. 

3. Establishment of effective means of assessing the accomplishment outcomes 
and results. 

Added to this, implicitly, is the use of the assessment results to improve the 

function of the institution or program. (Nichols, 1991, p. 10-12) 

Davis also broadly defined the current meaning of assessment and evaluation by 
merging the forms of assessment and evaluation "...into a common effort to understand 
and judge the merit and worth of teaching and learning within a course, curriculum. 
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educational program, sequence of study, department, unit, or institution " (Davis in Stark 
& Thomas, 1994, p. 47). 



Forms of Evaluation or Assessment 

Gardner described five basic kinds of evaluation used in education: 

". . .(1) evaluation as professional judgment, (2) evaluation as measurement, (3) evaluation 
as the assessment of congruence between performance and objectives (or standards of 
performance), (4) decision-oriented evaluation, and (5) goal-ffee/responsive evaluation” 
(Gardner in Stark & Thomas, 1994, p. 8). 

Although, as Gardner explained, there are many more variations of these "...but the 
principal emphasis or focus of a particular effort will almost always be identifiable as 
belonging to one of these categories" (Gardner in Stark & Thomas, 1994, p. 8). 

The five basic kinds of evaluations are used throughout all forms of assessment as 
they are in the various forms of evaluation of institutional effectiveness found in current 
models. 

"Professional judgment," the first form of evaluation, is used in some of the most 
common educational evaluations, particularly tenure review, accreditation, national or 
private funding proposals and articles for publication (Gardner in Stark & Thomas, 1994, 
p. 9). 

"Evaluation as Measurement ... traditional (but narrow) simply equates evaluation 
to measurement. To evaluate means to measure results, effects, or performance using 
some type of formalized instrument which produces data that can be compared to some 
sort of standardized scale" (Gardner in Stark & Thomas, 1994, p. 10). 
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Gardner continued to explain that even experts "...such as Thorndike [16] and Ebel 
[6,7] admit that the evaluation involves a judgment of merit which extends beyond the 
collection of measurement data, but the focus of this approach is clearly on those data and 
the instruments used to collect them" (Gardner in Stark & Thomas, 1994, p. 10). The 
results of these studies could be compared using standardized scales or comparability of 
multiple uses of the same questionnaire. The Scholastic Aptitude Test and Graduate 
Record Exam are traditional normed referenced examples. Indicators of effectiveness 
used by educational institutions to measure outcomes usually include the number of 
degrees granted, transfer rates and vocational placement rates. "The expected outcome 
from the measurement type of evaluation is a number or set of numbers which can be 
compared and interpreted with reference to another set of numbers, or a generally 
accepted standard scale" (Gardner in Stark & Thomas, 1994, p. 11). 

The "assessment of congruence between performance and objectives" (Gardner in 
Stark & Thomas, 1994, p. 1 1) evolves from Tyler's foundation work in evaluation and the 
behavioral theories related to education. "Theories or methodologies that fall into this 
category basically define evaluation as the process of specifying or identifying goals, 
objectives or standards of performance; identifying or developing tools to measure 
performance; and comparing the measurement data collected with the previously identified 
objectives or standards to determine the degree or discrepancy or congruence which 
exists" (Gardner in Stark & Thomas, 1994, p. 1 1). The most common forms of 
evaluation in this category include "competency based education" where goal oriented 
topics exist (Gardner in Stark & Thomas, 1994, p. 11). 
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Scrivens stated "...that this type of evaluation process can play two basic kinds of 

roles: a formative role (evaluation used to improve an ongoing process or project by 

providing feedback to the administration in charge), and summative role (evaluation of a 

completed product). ...Formal evaluation methodologies are required if rational decisions 

are made as to whether the program or thing should be maintained, improved, expanded 

or terminated. ..." Gardner continued, "the intended results of an evaluation of this sort 

are judgment of worth regarding the institution, program, process, or thing based on 

interpreted comparisons between performance data and objectives (or standards of 

performance)” (Gardner in Stark & Thomas, 1994, p. 12). Some common forms of 

measures used in institutional effectiveness models include: transfer institution satisfaction 

with transfer students which implies the adequacy of their preparation; employer 

satisfaction with program graduates employed by them; and academic or occupational 

students satisfaction with their preparation for transfer or direct employment 

Decisions-oriented evaluation includes two models with numerous derivatives. 

[The] CIPP [Context, Input, Process, Product] Evaluation Model developed by the 
Phi Delta Kappa (PDK) National Study Committee on Evaluation and a model 
developed by the University of California at Los Angeles Center for The Study of 
Evaluation. The two models are practically identical in their essential 
characteristics; the CIPP model is based on a definition of evaluation as 'The 
process of delineating, obtaining, and providing useful information forjudging 
decision alternatives,' [15,p.40] whereas the UCLA definition is stated as 
'Evaluation is the process of ascertaining the decision areas of concern, selecting 
appropriate information, and collecting and analyzing information in order to 
report summary data useful to decision-makers in selecting among alternatives' 
[lp.107], (Gardner in Stark & Thomas, 1994, p. 12) 

Some current existing models applying these principles include WICHE/NCHEMS 
Costing and Data Management System (Gardner in Stark & Thomas, 1994, p. 13). The 
current institutional effectiveness models imply a relationship between evaluation and 




55 



41 



improving institutions although some assessments follow more closely some other forms 
of evaluation. 

What Gardner describes in decision oriented evaluation is closely aligned with the 
intent of current institutional effectiveness models: 

The desired outcomes of an evaluation of this kind are a continual 
exchange between evaluators and administrators regarding information 
needs associated with critical decisions, and a continuous flow of 
systematically collected, timely, and relevant information to satisfy those 
needs. Final interpretation of data is generally assumed to be the 
responsibility of the administrator(s) served, as, for example, when the final 
report on grade statistics is placed in the hands of the academic vice- 
president who is looking for evidence of grade inflation. The evaluator’s 
interpretive skills are brought to bear more on issues related to data 
reduction and extraction (synthesis) than on meaning as it relates to 
decisions affecting institutional policy. (Gardner in Stark & Thomas, 1994, 
p. 14) 

Goal-free/Responsive Evaluation model is described as an approach used . .to 
discover and judge actual effects without regard to what the effects were supposed to be" 
(Gardner in Stark & Thomas, 1994, p. 14). This particular model, which follows the 
qualitative paradigm and general methodological prerequisites, has generally not been 
suggested by institutional effectiveness models. This approach is sometimes suggested for 
use at the program or course level. As Gardner points out, this might be appropriate in 
some program evaluation such as the arts. "In effect, by including informal investigation 
and personal testimony as valid tools for evaluation, this approach may help legitimize 
methods that have had intuitive appeal in many circumstances but have previously received 
little formal support" (Gardner in Stark & Thomas, 1994, p. 16). 
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Application of Effectiveness Models and Indicators for Accreditation 
and Institutional Change and Improvement 

The National Alliance of Community and Technical Colleges (NACTC) and the 
National Center for Research in Vocational Education developed an institutional 
effectiveness model which became the foundation of subsequent models. The model 
provides a structure for colleges to address external demands for effectiveness as well as 
providing information about the performance of their stated mission (Grossman & 

Duncan, 1988, p. 3). The model responds to both external demands and internal needs of 
the community college in addressing its effectiveness. 

The NACTC . identified six areas of inquiry that they believed to be generic to 
the mission of community and technical colleges in America” (Grossman & Duncan, 

1988, p. 27). The six areas or issues included access and equity, employment preparation 
and placement, college and university transfer, economic development, college/community 
partnerships and cultural and cross cultural development (Grossman & Duncan, 1988, p. 
18-19). 

The NACTC then developed indicators based on the literature as a logical 
generation of indicators to measure each area and in turn develop an instrument to 
measure institutional effectiveness. This model and indicators became the foundation for 
many subsequent models and reports including the American Association of Community 
Colleges (AACC) Core Indicators Model which is the primary influence for this survey. 

The NACTC most of all believed that the six areas of the model with its 38 
indicators provided community and technical colleges the opportunity to express . .their 
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best and most relevant case to external interests while serving as an accurate self-portrait 
for their own issues” (Grossman & Duncan, 1988, p. 39). 

In 1992, the Council on Postsecondary Accreditation (COP A) members of the task 
force on Institutional Effectiveness developed several papers on issues related to the 
assessment of institutional effectiveness and assessment process of institutional 
accreditation. Peter Ewell analyzed many of the issues surrounding assessment and its use 
in accreditation and developed several recommendations for resolving inherent conflict 
between these issues. He recommended several principles for future policy consideration 
which appear to have been adopted in regional accreditation standards and models such as 
the AACC Core Indicators model. 

The principles are: 

• “Assessment should not constitute a ‘component’ of institutional accreditation, 
but should rather permeate the entire accreditation process” (Ewell, 1992, 

p. 24). 

• “The primary focus of assessment should be on effectiveness of the institution 
as a whole” (Ewell, 1992, p. 24). 

• “A required focus of assessment should be placed upon student learning” 
(Ewell, 1992, p. 24). 

• “The primary use of assessment evidence in self-study should be to indicate 
progress, not to supply definitive judgments about quality” (Ewell, 1992, 
p. 25). 

• “The primary focus of assessment should be the degree to which the process is 
directed and results in continuous improvement” (Ewell, 1992, p. 25). 
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These principles seem to permeate the current accreditation model and 
requirements as well as other effectiveness models. 

As early as 1 984, however, the Southern Association of Colleges and Schools 
(SACS) had developed an outcomes-oriented accreditation (Hudgins, 1993, p. 42). 

“Ready or not, community colleges must deal with institutional effectiveness as they seek 
to reaffirm their accreditation” (Hudgins, 1993, p. 42). This mandate by the first of the 
accrediting associations based its decision on the fact that quality in education is based on 
“...(1) the appropriateness of its objectives; (2) the effectiveness of the use of resources in 
pursuing these objectives; (3) the degree to which these objectives are achieved. Without 
a clear statement of what education is expected to provide, it is not possible to determine 
how good it is (in Southern Association of Colleges and Schools, 1989, p. 19.)” (Hudgins 
1993, p. 43). 

The three basic questions of effectiveness have been slightly revised but drive most 
current models of institutional effectiveness. The questions are: 

1 . What is the mission (business) of our college? 

2. What are the major results we expect from the achievement of the mission? 

3. What specific evidence are we willing to accept that these results have been 
achieved? (AACC, 1997, p. xii). 

Midlands Technical College has used the structure developed by the National Alliance of 
Community and Technical Colleges (NACTC) to answer the questions of how institutions 
go about developing and implementing a model for institutional effectiveness. 

The six steps developed by NACTC included. 
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1. Establish a strategic planning process; ... 

2. Articulation of a vision and mission; ... 

3. Operationalize the planning process; ... 

4. Develop an evaluation system, and includes [identifying critical areas of 
success]. Establish indicators and standards by which a college can evaluate its 
effectiveness in the identified critical areas. ...Determine mechanisms for 
documenting if the established standards are met; ... 

5. Using data to improve effectiveness; ... 

6. Communicate the results. (AACC, 1997, p. xii) 

The other important ingredients suggested for a successful model include: 
“...support of the president and trustees; ...Linkage of assessment to the college mission; 
...Involvement of all units of the college; ...Establishment of an organizational structure to 
monitor and report on effectiveness outcomes; ...Use assessment data for decision 
making. [And]... adequately fund assessment activities.” (AACC, 1997, p. xiii). 

Based on this outline for a successful institutional effectiveness model, 
organizations and institutions continued to debate the issue of institutional effectiveness 
and related assessment issues during the 1980s and 1990s. AACC in 1994 published a 
Special Report No. 4 from the Community College Roundtable that was the culmination 
of many of the discussions during the previous ten year period. The Roundtable included 
many of the leaders in the institutional effectiveness movement. In its recommendation, 
the Roundtable based its model of core indicators of effectiveness not only on the 
literature or from educational leaders in this movement, but also from discussions at the 
“...annual meetings of AACC, the Community College Consortium, and the League for 
Innovation in the Community College to sharpen the focus of the core indicators” 

(AACC, 1994, p. 1). 
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The thirteen core indicators were based on the six mission categories serving most 
community colleges. The Core Indicators Model of Institutional Effectiveness developed 
by AACC also served as the primary basis for the questionnaire used in this study. 

The Core Indicators Model of Institutional Effectiveness developed by AACC 
includes: 

Mission: Student progress 

Core Indicator 1 : Student Goal Attainment 
Core Indicator 2: Persistence (Fall to Fall) 

Core Indicator 3: Degree Completion Rate 

Mission: Career Preparation 

Core Indicator 4: Placement Rate in the Workforce 
Core Indicator 5: Employer Assessment of Students 

Mission: Transfer Preparation 

Core Indicator 6: Number and Rate Who Transfer 
Core Indicator 7: Performance After Transfer 



Mission: Developmental Education 

Core Indicator 8: Success in Subsequent, Related Coursework 



Mission: General Education 

Core Indicator 9: Demonstration of Critical Literacy Skills 
Core Indicator 10: Demonstration of Citizenship Skills 



Mission: Customized Education 

Core Indicator 11: Client Assessment of Programs and Services 



Mission: Community Development 

Core Indicator 12: Responsive to Community Needs 

Core Indicator 13: Participation Rate in Service Area (AACC, 1994) 



The models “...can serve as a guide for a college developing its institutional 
effectiveness process. Each college must personalize the process and develop a model 
that is relevant to its mission, service region and college values” (AACC, 1997, p. xiii). 
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The seven practical outcomes achieved by developing and implementing an 
institutional effectiveness model include: 

• Clarification of mission 

• improved use of resources 

• identification of priorities 

• improved performance 

• increased return on resources invested 

• enhanced reputation 

• energized work environment. (ACCC, 1997, p. XV) 



Arthur Cohen in an extensive review of institutional effectiveness studies suggests 
that . .the validity of the indicators is suggested by the extent to which they reflect the 
colleges’ basic missions; they provide for a continual report on institutional effects” 
(Cohen, 1994, p. 1). 

There are three important facets to current indicators used in institutional 
effectiveness models: (a) they are easily understood by the internal and external college 
community and display evidence that community colleges are doing a good job; (b) the 
data is reasonably available both in time and fiscal resources; and (c) the indicators are 
primarily for use by the college community in reviewing its success relative to its mission 
(Cohen, 1994, p. 2). 



College Assessment of Institutional Effectiveness in the 1990s 
In order to determine the state of institutional assessment, a review was conducted 
of thirty research projects completed in the early 1990s. The projects were classified in 
several ways. First they were classified by facts: was the research theory or model 
development; was it applied research; or was the research centered on a single or multiple 
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institution. Second, the purpose of the assessment research was determined and classified 
by the purpose of the research which included compliance, persuading, monitoring, 
improving or judging. Third, the types of indicators assessed in the research were 
determined. This was important since the current focus of institutional effectiveness is 
generally focused on outcomes, even though other types of indicators are used. Other 
important indicators include input, process, contextual, return on investment, and 
customer satisfaction. 

The review of research projects was important to this study since it portrays the 
actual practices or behavior taking place in education during the early 1990s transition 
period. Although this is not intended to be a statistically quantifiable analysis, it does lend 
weight and description to the landscape evolving in education during this period. By 
1990, all regional accreditation associations had adopted assessment requirements aimed 
at outcomes and improved institutional effectiveness. This change should have begun to 
affect community college and administrative behavior and is the focus of this study. 
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Evaluation of Current Assessments 



Table 1 

Evaluation of Current Assessment 



Category 


Percentage of Utilization 


(A) FACTS 




Theory or Model Development 


20% 


Applied Research 


87% 


Single Institution 


33% 


Multiple Institution 


67% 


(B) PURPOSE 




Compliance 


20% 


Persuading 


7% 


Monitoring 


40% 


Improving 


63% 


Judging 


17% 


(C) TYPE OF INDICATORS USED 




Input 


30% 


Process (Environment) 


40% 


Outcomes 


93% 


Contextual 


10% 


Return on Investment 


3% 


Customer Satisfaction 


37% 



See Appendix A for details. 



Note: Categories are not mutually exclusive but represent the possibility of multiple uses. 



The results of this analysis of institutional research articles of institutional 
assessment and institutional effectiveness indicate that the majority are primarily engaged 
in applied research, as one might expect with many projects involving more than one 
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The period of the early 1990s was still a time of defining institutional effectiveness 
and assessment as it relates to accreditation and other outside political forces which might 
account for the number of projects involving model development and multiple institutional 
involvement. 

The purpose of assessment was probably the most significant finding in the review 
of research projects and supports the purpose of regional accreditation requirements 
which in essence expects community colleges to adopt institutional effectiveness models. 
The intent of these requirements is to improve institutions and 63% of projects supported 
this intent. Monitoring a closely related concept was represented in 40% of projects. 

The types of indicators mandated for use in determining institutional effectiveness 
and required by accreditation has moved generally from the input side of the equation 
towards one of outcomes, although other types still have merit for certain purposes. 
Colleges have clearly heard the call for outcomes based research, and this was represented 
in 93% of projects. Process indicators, also important to many activities in colleges, were 
represented in 40% of projects with customer satisfaction represented in 37% of projects. 

Since regional accreditation standards had not yet fully required outcomes based 
measures until 1990, including those of the Northwest Association of Schools and 
Colleges, the reflection of 30% of projects involving input indicators is an expected 
holdover from earlier years. Contextual indicators were found in 10% of projects and 
return on investment in only 3% of the reviewed projects. These two indicators appear 
less in the institutional projects and are probably more common at the state or system level 
analysis. What is clear in this review of research projects is that most assessment is 
applied; that the majority are completing assessments for institutional improvement and 
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that most indicators used are either outcomes, process, customer satisfaction or 
combination of the same for determining the effectiveness of institutions. 

The move towards outcomes based regional accreditation models for determining 
institutional effectiveness and continuous improvement is the current state of accreditation 
self study requirements. This still leaves unanswered the question, how far have we come 
in this movement and are we really assessing outcomes in our community colleges as 
mandated by accreditation and in turn using the information to initiate institutional change. 

By 1997 most state goveming/coordinating agencies had assessment measures as a 
part of state policy or regulation. In 1997, Nisson completed a survey of all state 
governing or coordinating agencies and stated four findings in the Emerging Vision of 
Accountability for College Improvement. 

These findings included: 

• Two year colleges in half the states are being required by legislature or Board 
to address the following question: Are colleges providing the benefits they are 
meant to provide and are they doing it in an increasingly effective manner or 
increasingly efficient manner? 

• While improvement is implicit everywhere, Washington and Missouri are the 
only states with an explicit requirement to demonstrate improvement prior to 
release of some part of funding. 

• Clear everywhere that accreditation is not addressing this external concern. 

• Raise issue of whether system can show two faces (external reporting to 
address above need and internal efforts such as part of accreditation and 
outcomes assessment) without being two-faced by connecting these two 
efforts. (SBCTC, 1997, p. 1) 

The intent of this study is to determine how far community colleges, accredited 
since 1990 by the Northwest Association of Schools and Colleges, have come in the use of 
effectiveness indicators for initiating institutional change. 





52 



Chapter III: Methods and Procedures 
Purpose 

The purpose of this study is to examine in selected northwest community colleges 
that have gone through accreditation since 1990, the perceived importance of using 
effectiveness indicators for assessing institutional outcomes and investigate their 
congruence with actual or potential use for making institutional changes and 
improvements. 



Hypothesis 

There seems to be a positive relationship in many community colleges between the 
identification and use of institutional effectiveness indicators and institutional change and 
improvement. 



Design 

This . . quantitative study, consistent with the quantitative paradigm, is an inquiry 

into a social or human problem, based on testing a theory composed of variables, 

measured with numbers, and analyzed with statistical procedures, in order to determine 

whether the predictive generalizations of the theory hold true” (Cresswell, 1994, p. 2). 

Cresswell suggested considering “quantitative methods as consisting of two types: 

Experiments include true experiments with the random assignment of subjects to 
treatment of conditions and quasi experiments that use nonrandom designs 
(Keppel, 1991). Included within quasi experiments are single-subject designs. 
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Surveys include cross-sectional and longitudinal studies using questionnaires or 

structured interviews for data collection with the intent of generalizing from a 

sample to a population (Babbie, 1990). (Cresswell, 1994, p. 10-11) 

This study has been based upon a review of the research literature and on 
information collected in a survey using a cross-sectional and field-tested study of a 
purposely selected sample of administrators and their views about the perceived 
importance of using effectiveness indicators for assessing institutional outcomes and 
congruence with actual or potential use for making institutional changes and 
improvements. The intent of the study will be to generalize the information for this select 
population of community college administrators (Cresswell, 1994, p. 11). 

This investigative study is done in a descriptive style using simple descriptive 
statistics of relevant data to demonstrate the perceived importance of using institutional 
effectiveness indicators for making purposeful institutional change and improvement. 

General and Research Questions 

Two general questions were examined in the study. 

1 . What kind of purposely selected effectiveness indicators are suggested by the 
accreditation process and related literature, and can be used to base 
institutional effectiveness assessment? 

2. What does the research literature have to say about the use of effectiveness 
indicators in assessment practices and for making institutional change? 

Six research questions were examined in the study. 

1 . Do community college administrators believe they should use effectiveness 
indicators for initiating institutional change? 
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2. Since the last community college regional accreditation (1990 or later) have 
community college administrators used effectiveness indicators for initiating 
institutional change? 

3. Do community college administrators believe that the use of effectiveness 
indicators led to positive changes in their institutions? 

4. Is there a difference by community college administrative position in the 
perception about the perceived importance of using effectiveness indicators for 
initiating institutional change? 

5. Is there a difference by community college administrative position in the use of 
effectiveness indicators for initiating institutional change? 

6. Is there a difference by community college administrative position in the 
perception that the use of effectiveness indicators led to positive changes in the 
institution? 



The Study 

A survey was conducted with purposely selected individuals at each institution 
including the president, primary instruction administrator, primary student services 
administrator, and primary business affairs administrator in northwest community colleges 
that have completed an outcomes based accreditation self study since 1990. 

The results of this survey have been analyzed by administrative position to 
compare and contrast the perceptions and use of institutional effectiveness indicators in 
community colleges and the application of those results by administrators in making 
purposeful institutional change and improvement. The survey did not attempt to define 
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which effectiveness indicators were most useful for making institutional changes or 
improvements or which strategies might best apply. There is no attempt in this study to 
determine a cause and effect relationship to actual changes and improvements. 

Data Analysis 

Simple descriptive statistics have been used to analyze the perceived importance of 
institutional effectiveness indicators and whether they were perceived to have potential use 
for making purposeful institutional change and improvement. 

Dependent Variables 

Three dependent variables were examined in the study: 

1 . The extent to which colleges should use effectiveness indicators for making 
institutional changes was measured by the responses of community college 
administrators responding to the belief that they should use indicators to 
initiate changes at their respective community college. 

2. The extent to which colleges are using effectiveness indicators for initiating 
institutional changes was measured by the response of administrators as to 
their use of indicators to initiate change at their respective community 
colleges. 

3. The extent to which they believe the use of effectiveness indicators led to 
positive changes at their community colleges was measured by the number of 
positive responses to the Likert type scale by community college 
administrators. 
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Survey 

A survey form of educational research was used to conduct the study through the 
use of a mailed questionnaire. In education . . survey research utilizes a variety of 
instruments and methods to study relationships, effects of treatments, longitudinal 
changes, and comparisons between groups” (Borg & Gall, 1989, p. 417). 

The population studied was the entire administrative population as defined in those 
institutions meeting the study criteria. The study in this case is called a census since the 
entire population meeting the criteria was included (Borg & Gall, 1989, p. 418). 

The questionnaire measured both perceptions and behavior about an educational 
practice, in the case of the study the use of effectiveness indicators for the purpose of 
initiating institutional change and improvement. Since this involves attitude, in this case a 
perception about a benefit of using effectiveness indicators, it used . .a number of items 
(usually at least 10) in order to obtain a reasonable picture of the attitude concerned” 
(Borg & Gall, 1989, p. 432). 

When measuring a person’s attitude it is important to investigate during the 
“...pretest whether the sample of subjects has sufficient knowledge and understanding to 
express a meaningful opinion about the particular topic” (Borg & Gall, 1989, p. 432). 

The pretest sample therefore used two institutions currently completing full accreditation 
studies and subject to visitation during 1997. The assumption was that given the current 
accreditation guidelines and requirement on assessment they should be knowledgeable 
about assessment and the implied changes which can be initiated as a result of the process. 

The implication of these findings is that the respondents’ knowledge and 
expertise is an important factor in interpreting attitude data. Therefore, when 
planning a questionnaire survey involving attitude measurement you should 
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investigate respondents’ familiarity with each attitude object covered in the survey. 
One technique for doing this is to administer an information test to a small sample 
of respondents similar to those to be queried in the main study, to determine 
whether they are capable of expressing an informed opinion about the persons, 
organizations, or educational practices mentioned in the attitude or opinion items. 
Another strategy often used is to include several information questions at the 
beginning of an attitude questionnaire to screen out respondents who display little 
or no knowledge of the attitude object. (Borg & Gall, 1989, p. 433) 

The pretest and survey assured a prior knowledge about the probable familiarity 

with the educational practice since it is required by accreditation during the selected 

period of the study and they as administrators in these institutions should have knowledge 

about the educational practice or use of effectiveness indicators being the object of the 

study (Borg & Gall, 1989, p. 433). If it was not a requirement of accreditation, which 

they had completed during this period, and as part of accreditation they should use 

effectiveness indicators for internal planning and decision making, then it should be an 

integrated activity of the institution. If this were not the case, prior knowledge could not 

be assumed and the survey would have used a threshold question to screen out 

respondents unfamiliar with the educational practice as suggested by Borg and Gall, 1989. 

In the pretest, the questionnaire was administered to a sample of administrators 

identified for use in the study in order to improve the meaning and understanding of the 

questions. The pretest population sample, as suggested by William Belson in Borg & Gall, 

1989, was asked to “repeat their understanding of the meaning of the question in their 

own words. Questions [were] revised and retested until they [were] understood by all or 

most members of the pretest sample” (Borg & Gall, 1989, p. 435). The process was the 

same as planned for the actual questionnaire process ( Borg & Gall, 1989, p. 435). The 

questionnaire was reviewed for comment feedback and responses to each item was 





explained to determine whether many were left blank or answered in unintended ways. 
This generally indicates misunderstanding by respondents (Borg & Gall, 1989, p. 435- 
436). 
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Field Testing of Survey 

The questionnaire was pre-tested on appropriate personnel in two community 
colleges. Peninsula Community College and Pierce Community College. 

Pretests was defined by Babbie as “. . . initial tests of one or more aspects of the 
research design. Most commonly, this has meant the administration of a draft 
questionnaire to a group of subjects, but the concept of pre-testing is more broadly 
applicable” (Babbie, 1990, p. 220). Pretests can include pre-testing the research 
instrument, data collection, data processing and analysis. Babbie encourages the use of a 
“. . . pilot study - a miniaturized walk-through of the entire study from sampling to 
reporting” (Babbie, 1990, p. 226). Although the above process is admirable, this 
research stayed with the more traditional approach of administering and revising the 
questionnaire with an appropriate group of similar subjects. The early forms of the survey 
were routed to personnel doing institutional research and those using performance or 
effectiveness indicators as a means of examining institutional effectiveness. The survey 
was then pre-tested, or field tested, on two sets of community college administrators as 
defined in the study but who were not included in the study because they served at 
institutions accredited by the Northwest Association of Schools and Colleges prior to 
1990, they were however, currently completing a full accreditation study during the 1997 
year. Focus groups were conducted on the two groups of administrators at two 
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institutions using open-ended questions to clarify survey questions formatting and issues 
surrounding the questionnaire. 

The questions were modified for clarity and understanding, particularly for issues 
surrounding content validity. The survey was reviewed, revised and re-administered to the 
second group of administrators to verify the recommendation for improvement, clarity, 
and understanding from the first group. The survey in its revised form was then finalized 
and prepared for distribution. 



The Survey Process 

The survey questions were to be directed to specific administrators in community 
colleges accredited since 1990 by the Northwest Association of Schools and Colleges. In 
order to be sure the survey was addressed to the correct person and position at each 
community college, a request was sent to each personnel director to obtain the correct 
names, titles and addresses for each selected administrative position. This list became the 
official mailing list for the survey. 

The survey followed both the development and mailing suggestions of Salant and 
Dillman, 1994, in order to achieve a good response rate. The questionnaire was 
developed as a confidential questionnaire, coded to insure accuracy by administrative 
position as well as institution and to allow for subsequent follow-up letters. 

A pre survey letter was sent to each selected person one week prior to the mailing 
of the survey. The survey was then mailed with a cover letter, questionnaire and return 
self addressed envelope. A third letter, sent a week later, thanked participants for their 
assistance and reminding those who had not returned their survey to please do so. (See 
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Appendix B). The fourth letter was sent approximately three weeks from the first survey 
mailing with a new cover letter, questionnaire and return addressed envelope, requesting a 
response and thanking them for their assistance (Salant & Dillman, 1994, p. 138). 

The survey process worked as expected with 148 questionnaires returned, or 
63.9% from the 232 originally mailed to 59 selected institutions. The number of valid 
surveys remaining after matching codes with returned responses reduced the original 148 
returned surveys to 128 or 55.2% valid survey responses. This was adjusted by 7 for 
administrators who represent two categories. These 135 respondents became the 
database for the analysis of the research questions. 

The Survey Analysis 

Descriptive analysis was used to examine questions one, two and three. 

Frequency distribution and mean scores were used to determine whether respondents 
believe indicators of effectiveness should be used, are used and the belief that the use led 
to positive change at their community colleges. 

The perception of value or should be and actual use, and whether the use led to 
positive change, was evaluated through the use of descriptive statistics based the 
summation of responses within the six categorical areas: student progression indicators of 
effectiveness; internal indicators; general education indicators; external indicators, career 
and customized education; external indicators, community development; and student or 
client satisfaction indicators. The extent to which administrators valued and used 
indicators or should use, were used, and whether the use led to positive change, were rank 
ordered by the mean scores and percentage of positive responses for each categorical area. 
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Each categorical area was further analyzed to determine which appeared to have the most 
and least support for initiating institutional change. 

Question four: Is there a difference by administrative position in the belief that 
they should use effectiveness indicators for initiating institutional change and 
improvement? Question four was further analyzed by testing the null hypothesis: There is 
no significant difference between community college presidents, primary instructional 
administrators, primary student services administrators and primary business affairs 
administrators in the belief that they should use effectiveness indicators to initiate change 
in their institutions. The question was analyzed by using the one-way analysis of variance, 
ANOVA, with a fixed design to determine if any significant differences existed between 
the mean scores of each administrative position. The extent to which they differ was 
analyzed using each dependent variable “should use” by each effectiveness category and 
each administrative position as the independent variable. To determine which group 
means were different the post hoc Boneferroni multiple comparisons test was used and 
based on a .05 level of significance. 

Question five: Is there a difference by administrative position in the use of 
effectiveness indicators for initiating institutional change and improvement? The question 
was analyzed by testing the null hypothesis: There is no significant difference between 
community college presidents, primary instructional administrators, primary student 
services administrators and primary business affairs administrators in the use of 
effectiveness indicators for initiating institutional change. The question was examined by 
using the one-way analysis of variance, ANOVA, with a fixed design to determine if a 
significant difference existed between the mean scores of each administrative position. 
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The extent to which they differ was analyzed by using the dependent variable are using by 
each effectiveness category and each administrative position as the independent variable. 
To determine which group means were different a post hoc Boneferroni multiple 
comparison test was used and based on a .05 level of significance. 

Question six: Is there a difference, by administrative position, in the belief that the 
use of effectiveness indicators led to positive change in the institution? The question was 
analyzed by testing the null hypothesis: There is no significant difference between 
community college presidents, primary instructional administrators, primary student 
services administrators and primary business affairs administrators in the belief that the use 
of effectiveness indicators led to institutional change. The question was analyzed by using 
One-Way Analysis of Variance, ANOVA, to determine if significant differences existed 
between the mean score of each administrative position. The extent to which they differ 
was analyzed by using the dependent variable perception of use led to change by each 
effectiveness category and each administrative position as the independent variable. To 
determine which group means were difference, the post hoc Boneferroni multiple 
comparison test was used and based on a .05 level of significance. 
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Chapter IV: Presentation and Analysis of Data 



The purpose of this study was to examine the perceived importance of using 
effectiveness indicators for assessing institutional outcomes by primary administrators in 
selected northwest community colleges that have undergone accreditation since 1990, and 
to investigate their congruence with actual or potential use for making institutional 
changes and improvements. This chapter presents information collected through a survey 
method and analyzed around eight questions. General questions one and two are related 
to the literature review and will not be reiterated in this section. Research questions 1-6 
are dependent on the information received and analyzed in the survey of primary 
administrators in northwest community colleges accredited since 1990, including 
presidents, primary instructional administrators, primary student services administrators 
and primary business affairs administrators. These are subsequently referred to as 
presidents, instructional administrators, student services administrators and business affairs 
administrators. 

This investigative study was done using descriptive statistics of relevant data to 
demonstrate the perceived importance of using institutional effectiveness indicators for 
making purposeful change in community colleges. The analysis provides a 
multidimensional examination of the attitudes and behaviors of primary administrators in 
northwest community colleges about the perceived importance of using effectiveness 
indicators, the actual use of effectiveness indicators and the perceived results of using 
effectiveness indicators as a means for making institutional change in their respective 
community colleges. 
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The analysis is presented in the order of the research questions with a short 
summary at the beginning of each question and an extensive summary at the conclusion of 
the chapter. 



Context of Study 

The survey was sent to 232 recipients. The survey sample represented a cross 
section of both institutions and primary administrators in northwest community colleges. 
Recipients returned 148 surveys or 63.8%. After removing all possible errors, particularly 
those who were not targeted by the survey, 128 valid returned surveys, or 55.2%, were 
used in the analysis, plus 7 duplicated surveys where the primary administrator represented 
both instruction and student services. The data was replicated for both positions to add 
equal weight to each primary position. The return rates for presidents was 25 or 42.4%; 
instructional administrators was 45 or 76.3%; student services administrators was 43 or 
72.9%; and business affairs administrators was 22 or 37.3%. 

I ( 

Valid responses were received from various size institutions: 63 participants or 
46.7% from institutions with 1-2499 FTE’s; 41 or 30.4% from institutions with 2500- 
4999 FTE’s; 22 or 16.3% from institutions with 5000-7499 FTE’s; 2 or 1.59% from 
institutions with 7500-9999 FTE’s; and 7 or 5.2% from institutions with 10,000 or more 
FTE’s. 

The primary administrators represented institutions from a variety of rural-urban 
settings as classified by each respondent. This population mix included: rural populations 
represented 69 participants or 51.1% of valid responses; suburban populations represented 
41 or 30.4% of valid responses; and colleges serving a mostly or all urban population 
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represented 25 or 18.5% of valid responses. Analysis of cross tabulations of 
administrative position and rural-urban mix indicated valid responses representing each 
administrative position at community colleges rated as rural, suburban or urban. 

Valid responses were also received from primary administrators representing each 
position in community colleges by FTE size from 1-7499. In FTE categories 7500-9999 
three administrative positions were not represented and in community colleges larger than 
10,000, business affairs administrators were not represented. 

This simple descriptive analysis provides the context or background which 
represents the northwest community college environment and establishes the context for 
the analysis of each question. 

Survey Categories and Indicator Measures 

The survey was based on responses to the following six categories of effectiveness 
indicators containing 27 different suggested measures, primarily derived from AACC Core 
Indicators Model, with the exception of Category 3: General Education and Category 6: 
Student Satisfaction, which were both derived from literature review and institutional 
practice. The following categories and indicators are those represented in the study. 
Category 1 : Student Progression Indicators 

1 . Student goal attainment. 

la. Measure: Number of students who leave and report their goal for 
attending college was met. 

2. Student persistence (Fall to Fall). 

2a. Measure: Fall Quarter cohort who are still enrolled the following quarter 
and have not graduated. 




so 



2b. Measure: Retention rates of students from quarter to quarter 
(credit students only). 

3 . Program completion . 

3 a. Measure: Degree and certificate completion rates. 

3b. Measure: Entering degree seeking students who complete a 
degree over time. Information is updated annually. 

Category 2: Internal Indicators 

4. Number and rate who transfer. 

4a. Measure: Number of entering students by cohort in a degree 
program, completing at least twelve credits, who enroll within 2 
years at a 4 year college in a degree program. 

5. Performance after transfer. 

5 a. Measure: Number of community college students transferring 
compared to native students at 4 year institutions who complete 
courses with a “C” or better. 

6. Success of remedial or developmental students in subsequent 

related college level course work. 

6a. Measure: Proportion of entering students deficient in one or more 
basic skills (reading, writing and computation) who complete 
developmental work within one year and who complete their first 
college level course with a “C” or better. 

6b. Measure: Number of students who progress from GED, ESL and 
developmental courses to success in college level courses. 

Category 3: General Education Indicators 

7. Demonstration of general education outcomes. 

7a. Literacy skills 

Measure: Determined locally by your community college. 

7b. Exposure to and knowledge of the natural world (science). 

Measure: Determined locally by your community college. 

7c. Exposure to and knowledge of the arts and humanities (humanities). 
Measure: Determined locally by your community college. 

7d. Exposure to and knowledge of culture (social science). 

Measure: Determined locally by your community college. 
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7e. Cultural awareness (pluralism/diversity). 

Measure: Determined locally by your community college. 

7f. Computational skills. 

Measure: Determined locally by your community college. 

7g. Critical thinking and research skills. 

Measure: Determined locally by your community college. 

7h. Application of skills, such as learning into action, community 
service or service learning. 

Measure: Determined locally by your community college. 

Category 4: External Indicators: Career and Customized Education 

8. Placement rate in the work force. 

8a. Measure: Proportion of entering student cohort who achieve a 
marketable skill by completing three or more courses in an 
occupational program and who obtain employment in a field 
directly related to that skill within one year after leaving. 

9. Employer assessment of students. 

9a. Measure: Proportion of sampled regional employers in a given field 
who indicate students from your community college have skills 
equal to or better than other employees. 

10. Client/customer assessment of programs and services. 

10a. Measure: Survey, focus groups or interviews to determine 
satisfaction of your customers, such as business and industry, 
public or private customers, about your programs, activities or 
services. 

Category 5: External Indicators (Community Development) 

1 1 . Responsiveness to community. 

11a. Measure: Survey, focus groups or interviews to identify community 
needs, your responsiveness to those needs, and a demonstration 
that the college has met or satisfied those needs. 

12. Participation rate in service area. 

12a. Measure: Proportion of local population in service area who have 
participated in an organized college activity during the past year. 
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Category 6: Customer Satisfaction 

1 3 . Student on customer satisfaction. 

13 a. Measure: Survey student satisfaction with preparation for their 
current employment one year after completing their program. 

13b. Measure: Employer satisfaction with your former students as graduates. 

13c. Measure: University and four year college satisfaction with 
academic preparation of your transfer students. 

13d. Measure: Student satisfaction with teaching effectiveness. 

13e. Measure: Student satisfaction with Student Services. 



Research Question 1 

Do community college administrators believe they should use effectiveness indicators for 
initiating institutional change? 

Summary of Significant Findings for Question 1 

First, overwhelming support was given by administrators about the value of using 
effectiveness indicators to initiate change. Support ranged from 85.5% to 97.0%. 

Second, three categories of indicators were perceived above a 90.0% level by all 
respondents, to be important as a way to initiate change at their institutions. These 
categories included: Category 4: External Indicators, Career and Customized Education, 
Category 6: Customer Satisfaction, and Category 1: Student Progression Indicators. 
Third, presidents rated highest the value of using Category 6: Customer Satisfaction, at 
98.3%. Fourth, instructional administrators rated highest the value of using Category 4: 
External Indicators, Career and Customized Education, at 97.8%. Fifth, student services 
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administrators rated highest the value of using Category 6: Customer Satisfaction, the 
same as community college presidents, at 98.1%. Sixth, business affairs administrators 
rated highest the value of using Category 4: External Indicators, Career and Customized 
Education, at 98.6%. This was also the highest rating for any single category by an 
administrative group. 

Presentation and Analysis of Data 

The survey asked respondents to rate their perception of six categories of 
indicators represented by 27 different suggested measures shown above and answer yes or 
no to the questions. The “y es ” answers were scored with the value of one and “no” 
responses were scored with the value of two. The responses were then averaged and 
categorized by all respondents and by each administrative position; presidents, 
instructional administrators, student services administrators and business affairs 
administrators. 

In the calculation of mean scores, yes was coded “1” and no was coded “2”, 
therefore, the lower the mean score the more positive the answer. A mean score of 1 
would indicate a 100% yes response rate. 

The rank order of the six categories based on the highest percentage of “yes” 
responses or lowest mean scores is shown in Table 2. 
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Table 2 

Perception of Indicator: 
All respondents. 

Rank Order 



Rank order by “yes” 



Categoiy 


Indicator 


n 


Mean 


Yes 


No 






135 




n 


% 


n 


% 


















Category 4 


External Indicators: Career 
and Customized Education 


134.0 


1.03 


130.0 


97.0 


4.0 


3.0 


Category 6 


Customer Satisfaction 


130.8 


1.03 


126.4 


96.6 


4.4 


3.4 


Category 1 


Student Progression Indicators 


134.8 


1.09 


122.4 


90.8 


12.4 


9.2 


Category 2 


Internal Indicators 


124.7 


1.12 


109.5 


87.8 


15.2 


12.2 


Category 5 


External Indicators: 
Community Development 


135.0 


1.14 


116.0 


85.9 


19.0 


14.1 


Category 3 


General Education Indicators 


131.1 


1.14 


112.1 


85.5 


19.0 


14.5 



Table 2 depicts an overwhelming belief by all primary administrators that they should use 
effectiveness indicators as a means to initiate change at their community colleges. The yes 
responses ranged from a low mean score of 1. 14 and 85.5% for Category 3: General 
Education Indicators to a mean score of 1.03 and 97.0% for Category 4: External 
Indicators: Career and Customized Education. 



The same analysis by mean scores and percentage distribution was completed for 
each administrative position in the same manner as Table 2. Table 3 depicts the belief by 
presidents that they should use effectiveness indicators as a way to initiate change in their 
community colleges. 

The rank order of categories by the highest percentage of “yes” responses and the 
lowest mean scores is shown in Table 3. 
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Table 3 

Perception of Indicator: 
Presidents, 

Rank Order 



Category 


Indicator 


n 


Mean 


Yes 


No 






25 




n 


% 


n 


% 


















Category 6 


Customer Satisfaction 


24 


1.02 


23.6 


98.3 


0.4 


1.7 


Category 1 


Student Progression Indicators 


25 


1.02 


24.4 


97.6 


0.6 


2.4 


Category 4 


External Indicators: Career 
and Customized Education 


25 


1.04 


24.0 


76.0 


1.0 


4.0 


Category 3 


General Education Indicators 


24.7 


1.09 


22.4 


90.7 


2.3 


9.3 


Category 2 


Internal Indicators 


23 


1.11 


20.5 


89.1 


2.5 


10.9 


Category 5 


External Indicators: 
Community Development 


25 


1.16 


21.0 


84.0 


4.0 


16.0 



The Presidents’ mean scores and distribution of “yes” percentages indicate strong 
belief in the value of using effectiveness indicators to initiate change at their community 
colleges. The responses ranged from a low of 1. 16 mean and 84.0% yes in Category 5: 
External Indicators: Community Development, to a high of 1.02 mean and 98.3% “yes, is 
important” as a way to initiate change, in Category 6: Customer Satisfaction. 



Table 4 displays the belief by instructional administrators that they should use 
effectiveness indicators as a way to initiate change in their institutions. The responses are 
displayed in rank order of categories by the highest percentage of “yes” responses or 



lowest mean scores. 
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Table 4 

Perception of Indicator: 
Instructional Administrators, 
Rank Order 



Category 


Indicator 


n 


Mean 


Yes 


No 






45 




n 


% 


n 


% 


















Category 4 


External Indicators: Career 
and Customized Education 


45.0 


1.02 


44.0 


97.8 


1.0 


2.2 


Category 6 


Customer Satisfaction 


43.6 


1.02 


, 42.6 : 


97.7 


1.0 


2.3 


Category 1 


Student Progression Indicators 


44.8 


1.10 


40.2 


89.7 


4.6 


10.3 


Category 2 


Internal Indicators 


41.5 


1.10 


37.2 


89.6 


4.3 


10.4 


Category 5 


External Indicators: 
Community Development 


45.0 


1.13 


39.0 


86.7 


6.0 


13.3 


Category 3 


General Education Indicators 


44.0 


1.17 


36.7 


83.4 


7.3 


16.6 



Instructional administrators, similar to the presidents, indicate a belief that they 
should use effectiveness indicators to initiate change at their institutions. The mean score 
and percentage of “yes” responses ranged from a low of 1.17 and 83.4% “yes, is 
important” in Category 3: General Education Indicators, to 1.02 and 97.8% “yes” 
responses in Category 4: External Indicators: Career and Customized Education. 



Table 5 displays the belief by student services administrators that they should use 
effectiveness indicators to initiate change at their institutions in rank order of categories by 
the highest percentage of “yes” responses and the lowest mean scores. 
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Table 5 

Perception of Indicator: 

Student Services Administrators, 
Rank Order 



Category 


Indicator 


n 


Mean 


Yes 


No 






43 




n 


% 


n 


% 


















Category 6 


Customer Satisfaction 


41.4 


1.02 


40.6 


98 1 


.8 


1.9 


Categoiy 4 


External Indicators: Career 
and Customized Education 


42.0 


1.04 


40.3 


96.0 


1.7 


4.0 


Category 1 


Student Progression Indicators 


43.0 


1.11 


38.2 


88.8 


4.8 


11.2 


Category 2 


Internal Indicators 


39.2 


1.14 


33.7 


86.0 


5.5 


14.0 


Categoiy 5 


External Indicators: 
Community Development 


43.0 


1.15 


36.5 


84.9 


6.5 


15.1 


Category 3 


General Education Indicators 


40.6 


1.15 


34.3 


84.5 


6.3 


15.5 



The yes responses ranged form a low in Category 3: General Education Indicators 
of 1.15 mean and 84.5% “yes, is important” to a high in Category 6: Customer 
Satisfaction of 1.02 mean and 98.1% “yes is important.” Student services administrators 
display very strong general belief that they should use effectiveness indicators as a way to 
initiate change at their community colleges. 



Table 6 displays the perception of business affairs administrators for the use of 
effectiveness indicators to initiate change at their community colleges in rank order of 
categories from the highest percentage of “yes” responses and the lowest mean scores. 
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Table 6 

Perception of Indicator: 
Business Affairs Administrators, 
Rank Order 



Category 


Indicator 


n 


Mean 


Yes 


No 






22 




n 


% 


n 


% 


















Category 4 


External Indicators: Career 
and Customized Education 


22.0 


1.01 


21.7 


98.6 


.3 


1.4 


Category 6 


Customer Satisfaction 


21.8 


1.10 


19.6 


89.9 


2.2 


10.1 


Category 5 


External Indicators: 
Community Development 


22.0 


1.11 


19.5 


88.6 


2.5 


11.4 


Category 1 


Student Progression Indicators 


22.0 


1.11 


19.6 


89.1 


2.4 


10.9 


Category 3 


General Education Indicators 


21.7 


1.14 


18.6 


85.7 


3.1 


14.3 


Category 2 


Internal Indicators 


21.0 


1.14 


18.0 


85.7 


3.0 


14.3 



Business affairs administrators, similar to other administrative positions indicate 
belief that they should use effectiveness indicators to initiate change at their institutions. 
The belief ranged from a low in Category 2: Internal Indicators, with a 1. 14 mean and 
85.7% “yes, is important” to a high in Category 4: External Indicators: Career and 
Customized Education with a mean of 1.01 and 98.6% “yes, is important” as a means to 
initiate change. 

Table 7 shows that administrators responding to the survey indicated strong belief 
that they should use effectiveness indicators as a way to initiate change at their institutions 
as indicated by the mean scores being very close to 1 .0 which would indicate complete 




positive response. 
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Table 7 

Perception of Indicator: 
Comparison of Mean Scores 
by Administrative Position, 
Categorical Order 



Category 


Indicator 


All 


Presidents 


Instruction 


Student 

Services 


Business 

Affairs 






n=135 


n=25 


n=45 


n=43 


n=22 
















Category 1 


Student Progression 
Indicators 


1.09 


1.02 


1.10 


1.11 


1.11 


Category 2 


Internal Indicators 


1.12 


1.11 


1.10 


1.14 


1.14 


Category 3 


General Education 
Indicators 


1.14 


1.09 


1.17 


1.15 


1.14 


Category 4 


External Indicators: 
Career and Customized 
Education 


1.03 


1.04 


: 1.02 


1.04 


1.01 


Category 5 


External Indicators: 

Community 

Development 


1.14 


1.16 


1.13 


1.15 


1.11 


Category 6 


Customer Satisfaction 


1.03 


1.02 


1.02 


1.02 


1.10 



Note: Shaded areas indicate categories with the most positive mean score or scores. 



Figure 3 shows the percentage of yes responses by each administrative position 
ranging from 89.7% to 92.6% positive support. 
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Figure 3 

Percentage of Administrators Believing 
They Should Use Effectiveness Indicators 
to Initiate Change 
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Research Question 2 

Since the last community college regional accreditation (1990 or later) have 
community college administrators used effectiveness indicators for initiating change and 
improvement? 



Summary of Significant Findings for Question 2 

First, all respondents rated four categories of indicators above the 50% level of 
usage. These categories included: Category 4: External Indicators, Career and 
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Customized Education; Category 6: Customer Satisfaction; Category 5: External 
Indicators, Community Development; and Category 1 : Student Progression Indicators. 
Second, presidents indicated they used Category 6: Customer Satisfaction, the most at 
72.0%, as a way to initiate change. Third, instructional administrators indicated they used 
Category 4: External Indicators, Career and Customized Education, the most at 69.7%, as 
a way to initiate change. Fourth, student services administrators indicated they used 
Category 6: Customer Satisfaction Indicators, the most at 63.9%, as a way to initiate 
change. Fifth, business affairs administrators indicated they used Category 4: External 
Indicators, Career and Customized Education Indicators, the most at 40.9%, as a way to 
initiate change. Sixth, the least used category was Category 3: General Education 
Indicators at 30.6%. Seventh, indicators were used most for input to decision making 
processes involving planning, improving services, budgeting, accreditation, and program 
review. Eighth, indicators were used at low levels for decisions regarding improved 
learning ranked number seven and community accountability ranked number eleven. 

Presentation and Analysis of Data 

The review of literature clearly indicated the need to establish effectiveness 
indicators as a means of initiating changes in community colleges. This is reinforced in all 
regional accrediting standards including the Northwest Association of Schools and 
Colleges since 1990. One intent of the survey was to investigate whether administrators 
were in fact using indicators suggested from national models and accreditation standards 
as a means to initiate change at their institutions. 
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The analysis of question 3 established the belief by primary administrators that they 
should use effectiveness indicators to initiate change. Question 4 determines if 
administrators are actually using effectiveness indicators to initiate change. The following 
detailed analysis reflects the indicated use of effectiveness indicators as a way to initiate 
change at community colleges. The information is displayed for all respondents and 
separately for each primary administrative group. 

Table 8 displays the categories of effectiveness indicators in rank order of mean 
scores and frequency of use by all respondents. The mean scores in the table were 
calculated with “yes” equal to one and “no” equal to two. Categories with lower mean 
scores indicate higher usage rates by administrators. 



Table 8 

Use of Indicators: 
All Respondents, 
Rank Order 



Category 


Indicator 


n 


Mean 


Yes 


No 






135 




n 


% 


n 


% 


















Category 4 


External Indicators: Career 
and Customized Education 


124.0 


1.40 


74.7 


60.2 


49.3 


39.8 


Category 6 


Customer Satisfaction 


121.4 


1.41 


71.4 


58.8 


50.0 


41.2 


Category 5 


External Indicators: 
Community Development 


117.5 


[1.49 


59.5 


50.6 


58.0 


49.4 


Category 1 


Student Progression Indicators 


123.2 


1.51 


60.4 


49.0 


62.8 


51.0 


Category 2 


Internal Indicators 


115.0 


1.58 


48.7 


42.3 


66.3 


57.7 


Category 3 


General Education Indicators 


121.2 


1.75 


30.6 


25.2 


90.6 


74.8 
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The mean scores and frequency of use ranges from a low mean of 1.75 and 30.6% 
use for General Education Indicators to a high of 1 .40 or 60.2% use for External 
Indicators, Career and Customized Education. Three indicators. External Indicators: 
Career and Customized Education, Customer Satisfaction and External Indicators: 
Community Development, were used by 50% or more of the administrative respondents. 
The remainder of categories were used below the 50% level as a means to initiate change 
at their community colleges. 



Table 9 displays the categories of effectiveness indicators in rank order of mean 
scores and frequency of use by community college presidents. 



Table 9 

Use of Indicators: 
Presidents, 

Rank Order 



Category 


Indicator 


n 


Mean 


Yes ! 


No 










n 


% 


n 


% 


















Category 6 


Customer Satisfaction 


23.6 


1.28 


17.0 


72.0 


6.6 


28.0 


Categoiy 4 


External Indicators: Career 
and Customized Education 


24.3 


1.31 


16.7 


68.7 


7.6 


31.3 


Category 5 


External Indicators: 
Community Development 


23.0 


1.37 


14.5 


63.0 


8.5 


37.0 


Category 1 


Student Progression Indicators 


24.2 


1.42 


14.0 


57.9 


10.2 


42.1 


Category 2 


Internal Indicators 


22.0 


1.50 


11.0 


50.0 


11.0 


50.0 


Category 3 


General Education Indicators 


24.5 


1.64 


8.9 


36.3 


15.6 


63.7 



The frequency of using effectiveness indicators and mean scores range from a low 
mean score of 1 .64 or 36.6% for Category 3: General Education Indicators to a high mean 
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score of 1.28 and 72.0% for Category 6: Customer Satisfaction. Indicators with a use 
exceeding 50% by presidents included five of six categories of effectiveness indicators. 



Tables 10 displays the categories of effectiveness indicators in rank order of mean 
scores and frequency of use by community college instructional administrators. 



Table 10 

Use of Indicators: 
Instructional Administrators, 
Rank Order 



Category 


Indicator 


n 


Mean 


Yes 


No 






Z 45 




n 


% 


n 


% 


















Category 4 


External Indicators: Career 
and Customized Education 


40.6 


1.30 


28.3 


69.7 


12.3 


30.3 


Category 6 


Customer Satisfaction 


40.0 


1.38 


24.6 


61.5 


15.4 


38.5 


Category 5 


External Indicators: 
Community Development 


39.0 


1.42 


22.5 


57.7 


16.5 


42.3 


Category 1 


Student Progression Indicators 


38.8 


1.46 


20.8 


53.6 


18.0 


48.4 


Category 2 


Internal Indicators 


37.2 


1.48 


19.2 


51.6 


18.0 


48.4 


Category 3 


General Education Indicators 


37.5 


1.73 


10.2 


27.2 


27.3 


72.8 



The mean scores and frequency of use by instructional administrators range from a 
low mean score of 1.73 and 27.2% use in Category 3: General Education Indicators, to a 
high mean score of 1.30 and 69.7% in Category 4: External Indicators, Career and 
Customized Education. The effectiveness indicators used by instructional administrators 
above the 50% level include all categories of indicators except Category 3: General 
Education Indicators. 
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Table 1 1 displays the categories of effectiveness indicators in rank order of mean 
scores and frequency of use by student services administrators. 



Table 1 1 

Use of Indicators: 

Student Services Administrators, 
Rank Order 



Category 


Indicator 


n 


Mean 


Yes 


No 






43 




n 


% 


n 


% 


















Category 6 


Customer Satisfaction 


38.2 


1.36 


24.4 | 


63.9 


13.8 


36.1 


Category 4 


External Indicators: Career 
and Customized Education 


38.6 


1.45 


21.3 


55.2 


17.3 


44.8 


Category 1 


Student Progression Indicators 


39.4 


1.47 


20.8 


52.8 


18.6 


47.2 


Category 5 


External Indicators: 
Community Development 


35.5 


1.52 


17.0 


47.9 


18.5 


52.1 


Category 2 


Internal Indicators 


36.2 


1.60 


14.5 


40.0 


21.7 


60.0 


Category 3 


General Education Indicators 


37.5 


1.75 


9.2 


24.5 


28.3 


75.5 



The mean scores and frequency of use by student services administrators range 
from a low of 24.5% and a mean of 1.75 for Category 3: General Education Indicators to 
a high of 63.9% and a mean of 1 .36 for Category 6: Customer Satisfaction. Student 
services administrators used three categories of indicators above a 50% level. These 
include: Category 6: Customer Satisfaction; Category 4: External Indicators, Career and 
Customized Education; and Category 1 : Student Progression Indicators. 

Table 12 displays the categories of effectiveness indicators in rank order of mean 
scores and frequency of use by business affairs administrators. 
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Table 12 

Use of Indicators: 

Business Affairs Administrators, 
Rank Order 



Category 


Indicator 


n 


Mean 


Yes 


t No 






22 




n 


% 


n 


% 


















Category 4 


External Indicators: Career 
and Customized Education 


20.3 


1.59 


8.3 


40.9 


12.0 


59.1 


Category 6 


Customer Satisfaction 


19.6 


1.72 


5.4 


27.6 


14.2 


72.4 


Category 5 


External Indicators: 
Community Development 


20.0 


1.72 


5.5 


27.5 


14.5 


72.5 


Category 1 


Student Progression Indicators 


20.8 


1.77 


4.8 


23.1 


16.0 


76.9 


Category 2 


Internal Indicators 


19.5 


1.79 


4.0 


20.5 


15.5 


79.5 


Category 3 


General Education Indicators 


21.7 


1.90 


2.2 


10.1 


19.5 


89.9 



The mean scores and frequency of use for business affairs administrators range 
from a low of 10. 1% and a mean of 1 .90 for Category 3: General Education Indicators, to 
a high of 40.9% and a mean score of 1.59 for Category 4: External Indicators, Career and 
Customized Education. The use of indicators by business affairs administrators were all 
below 50% use. This was the only administrative category which had no indicators used 
above the 50% level. 



Table 13 displays in categorical order the mean scores of using effectiveness 
indicators as a way to initiate change for all respondents and for each administrative 
position. 
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Table 13 
Use of Indicator: 
Comparison of Mean Scores 
by Administrative Position, 
Categorical Order 



Category 


Indicator 


All 


Presidents 


Instruction 


Student 

Services 


Business 

Affairs 






n=135 


n=25 


n=45 


n=43 


n=22 
















Category 1 


Student Progression 
Indicators 


1.51 


1.42 


1.46 


1.47 


1.77 


Category 2 


Internal Indicators 


1.58 


1.50 


1.48 


1.60 


1.79 


Category 3 


General Education 
Indicators 


1.75 


1.64 


1.73 


1.75 


1.90 


Category 4 


External Indicators: 
Career and Customized 
Education 


1.40 


1.31 


1.30 


1.45 


1.59 


Category 5 


External Indicators: 

Community 

Development 


1.49 


1.37 


1.42 


1.52 


1.72 


Category 6 


Customer Satisfaction 


1.41 


1.2$ 


1.38 


1.36 


1.72 



Note: Shaded areas indicate mean values for categories with the highest administrative 
use. 



The lowest mean values indicate highest administrative use of indicators. Category 
4: External Indicators, Career and Customized Education, ranked the highest by mean 
score. This category includes placement rate in the workforce, employer assessment of 
students and client/customer assessment of programs and services. Category 6: Customer 
Satisfaction, ranked second most used. Customer Satisfaction includes indicators such as, 
student satisfaction with preparation for their current employment one year after 
completing their program, employer satisfaction with former students as graduates, 
university and four year college satisfaction with academic preparation of transfer 
students, student satisfaction with teaching effectiveness, and student satisfaction with 
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student services. 
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Figure 4 displays the use of effectiveness indicators by each administrative position 
and shows a range of use from a low of 24.8% by business affairs administrators to a high 
of 58.0% by presidents. 



Figure 4 

Percentage of Administrators Using 
Effectiveness Indicators to Initiate Change 



Administrative 

Positions 



All Respondents 



Presidents 



Instructional Administrators 



Student Services Administrators 



Business Affairs Administrators 




0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0% 

Percentage "Using" Indicators 



The use of indicators as a way of initiating change was further analyzed by survey 
question 1 6. Have you in your area of responsibility used assessment information as 
input to the decision making process in any of the following areas ? The question 
included eleven areas and one open-ended response category. The eleven categories 
included planning, budgeting, program review, accreditation, quality improvement 




measures, improving services, improving learning, improving occupational programs, 
improving general education outcomes, legislative accountability, community 
accountability, and other. 

Table 14 depicts the use of indicators or assessment information as input to the 
specified decision making process by all respondents and each administrative group. 



Table 14 

Use of Indicators in 
Decision Making Process 
by Administrative Position, 
Categorical Order 



Decision Making Process 


All 


Presidents 


Instruction 


Student 

Services 


Business 

Affairs 


Planning 


1 


i 


1 


2 . 


1 


Budgeting 


3 


2 


5 


5' 


1 


Program review 


T " 


4" 


2 


” . 3 


6 


Accreditation 


4 


l 


3 


4 


■ 3 ■ 


Quality improvement measures 


6 


6 


7 


6 


3 


Improving services 


2 


3 


4; ■ ; . 


k;, 1 


2 


Improving learning 


7 


6 


3 


8 


9 


Improving occupational programs 


8 


5 


“T" 


10 


8 


Improving general education 
outcomes 


10 


8 


6 


11 


7 


Legislative accountability 


9 


7 


8 


7 


■ 4 


Community accountability 


n 


8 


9 


9 


5 


Other 


12 


9 


10 


12 


10 



Note: Shaded areas indicate categories that ranked in the top five by each administrative 
position. 



The use of indicators in certain decision making categories was almost the same 
for each administrative group, as indicated by the shading in Table 14. Indicators and/or 
assessment information were used the most in planning, budgeting, program review, 
accreditation and improving services. Others rating in the top five by one or more 



administrative groups included quality improvement measures, improving occupational 
programs, legislative accountability and community accountability. 
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Research Question 3 

Do community college administrators believe that the use of effectiveness indicators led 
to positive changes or improvements in their institutions ? 

Summary of Significant Findings for Question 3 

First, all respondents believed that Category 5: External Indicators, Community 
Development, was the indicator which led to the most positive change. Second, 
presidents believed Category 4: External Indicators, Career and Customized Education, 
was the indicator which led to the most positive change. Third, instructional, student 
services and business affairs administrators believed Category 5: External Indicators, 
Community Development, was the indicator which led to the most positive change. 

Fourth, all respondents rated two categories between positive and very positive. The two 
categories were Category 5: External Indicators, Community Development and Category 
4: External Indicators, Career and Customized Education, as the indicators which led to 
the most positive change. 

Presentation and Analysis of Data 

The purpose of this section is to examine the perception of primary administrators 
who have used effectiveness indicators and whether that use led to positive change at their 
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community colleges. The survey qualified this question about respondent’s perception to 
those administrators who used effectiveness indicators to initiate change at their institution 
as indicated in the survey. 

The information was captured using a 5 point Likert type scale ranging from very 
negative change to very positive change with an added “Don’t know,” which is considered 
an invalid response for the purpose of this study. 

The following tables summarize the frequency of responses and mean scores for all 
respondents and by each administrative position. In these tables the higher the mean score 
on the Likert type scale the more positive the perception about the use of indicators 
leading to positive change at their community colleges. 

Table 15 displays the categories of indicators, in rank order of positive mean 
scores, for all respondents and their perception about the use of effectiveness indicators 
leading to positive change at their community colleges. 
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Table 15 

Perception of Use and Change: 
All Respondents, 

Rank Order 



Cat. 


N 


Mean 


Very 

Negative 


Negative 


No Change 


Positive 


Very Positive 




135 




n 


% 


n 


% 


n 


% 


n 


% 


n 


% 




























5 


56.0 


1.20 


0 


0 


0 


0 


5.5 


9.8 


33.5 


59.8 


17.0 


30.4 


4 


73.7 


1.15 


0 


0 


.7 


1.0 


5.0 


6.8 


50.3 


68.2 


17.7 


24.0 


3 


29.4 


.97 


0 


0 


0 


0 


4.7 


16.0 


20.9 


71.1 


3.8 


12.9 


6 


68.4 


.95 


0 


0 


.2 


.3 


10.6 


15.5 


49.8 


72.8 


7.8 


11.4 


1 


56.4 


.79 


0 


0 


0 


0 


16.4 


29.1 


35.2 


62.4 


4.8 


8.5 


2 


45.0 


.79 


0 


0 


1.3 


2.9 


13.0 


28.9 


24.3 


54.0 


6.4 


14.2 



Category 5=Extemal Indicators: Community Development 
Category 4=Extemal Indicators: Career and Customized Education 
Category 3=General Education Indicators 
Category 6=Customer Satisfaction 
Category l=Student Progression Indicators 
Category 2=Intemal Indicators 



The perception of use and change ranged from a low mean score of .79 on a scale 
of -2 to +2, for Category 2: Internal Indicators, and Category 1: Student Progression 
Indicators, to a high of 1.20 for Category 5: External Indicators, Community 
Development. All categories of indicators were perceived to have led to positive change 
at their institutions. 



Table 16 displays the categories of indicators, in rank order of positive mean 



scores for presidents and their perception about the use of effectiveness indicators leading 
to positive changes at their community colleges. 
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Table 16 

Perception of Use and Change: 
Presidents, 

Rank Order 



Cat. 


N 


Mean 


Very 

Negative 


Negative 


No Change 


Positive 


Very Positive 




25 




n 


% 


n 


% 


n 


% 


n 


% 


n 


% 




























4 


16.0 


1.29 


0 


0 


0 


0 


1.7 


10.6 


8.0 


50.0 


6.3 


39.4 


5 


13.5 


1.26 


0 


0 


0 


0 


1.0 


7.4 


8.0 


59.3 


4.5 


33.3 


3 


8.2 


1.04 


0 


0 


0 


0 


.6 


7.3 


6.7 


81.7 


.9 


11.0 


2 


7.2 


.97 


0 


0 


0 


0 


1.8 


25.0 


3.8 


52.8 


1.6 


22.2 


6 


16.6 


.92 


0 


0 


.2 


1.2 


2.4 


14.5 


12.6 


75.9 


1.4 


8.4 


1 


12.4 


.87 


0 


0 


0 


0 


3.2 


25.8 


7.6 


61.3 


1.6 


12.9 



Category 4=Extemal Indicators: Career and Customized Education 

Category 5=Extemal Indicators: Community Development 

Category 3=General Education Indicators 

Category 2=Intemal Indicators 

Category 6=Customer Satisfaction 

Category l=Student Progression Indicators 



The perception of presidents about the use of effectiveness indicators leading to 
positive change can be seen in this rank order. The rank order of mean scores ranged 
from a low of .87 for Category 1: Student Progression Indicators, to a high of 1.29 for 
Category 4: External Indicators, Career and Customized Education. All indicators had 
positive mean scores with three categories: Category 4: External Indicators, Community 
Development; Category 5: External Indicators, Career and Customized Education; and 
Category 3: General Education Indicators, being rated between positive and very positive. 



Table 17 displays the categories of indicators, in rank order of positive mean 
scores, for instructional administrators and their perception about the use of effectiveness 
indicators leading to positive change at their community colleges. 
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Table 17 

Perception of Use and Change: 
Instructional Administrators, 
Rank Order 



Cat. 


N 


Mean 


Very 

Negative 


Negative 


No Change 


Positive 


Very Positive 




45 




n 


% 


n 


% 


n 


% 


n 


% 


n 


% 




























5 


20.0 


1.30 


0 


0 


0 


0 


1.5 


7.5 


11.0 


55.0 


7.5 


37.5 


4 


27.0 


1.17 


0 


0 


.3 


1.1 


.7 


2.6 


20.0 


74.1 


6.0 


22.2 


3 


10.2 


1.08 


0 


0 


0 


0 


1.5 


14.7 


6.4 


62.7 


2.3 


22.6 


6 


23.0 


1.01 


0 


0 


0 


0 


3.4 


14.8 


16.0 


69.6 


3.6 


15.6 


2 


18.7 


.90 


0 


0 


.5 


2.7 


4.5 


24.0 


10.0 


53.5 


3.7 


19.8 


1 


19.2 


.88 1 


0 


0 


0 


0 


4.4 


22.9 


12.6 


65.6 


2.2 


11.5 



Category 5=Extemal Indicators: Community Development 

Category 4=Extemal Indicators: Career and Customized Education 

Category 3=General Education Indicators 

Category 6=Customer Satisfaction 

Category 2=Intemal Indicators 

Category l=Student Progression Indicators 



The perception by instructional administrators about the use of effectiveness 
indicators leading to positive change at their institutions ranged from a low mean of .88 
for Category 1: Student Progression Indicators, to a high mean of 1.30 for Category 5: 
External Indicators, Community Development. Four of the six indicators mean scores 
were rated above 1.0, or positive to very positive. The positively rated indicators 
included: Category 5: External Indicators, Community Development; Category 4: External 
Indicators, Career and Customized Education; Category 3: General Education; and 
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Category 6: Customer Satisfaction. 
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Table 18 displays the categories of indicators, in rank order of positive mean 
scores, for student services administrators and their perception about the use of 
effectiveness indicators leading to positive change at their community colleges. 



Table 18 

Perception of Use and Change: 
Student Services Administrators, 
Rank Order 



Cat. 


N 


Mean 


Very 

Negative 


Negative 


No Change 


Positive 


Very Positive 




43 




n 


% 


n 


% 


n 


% 


n 


% 


n 


% 




























5 


17.0 


1.06 


0 


0 


0 


0 


3.0 


17.6 


10.0 


58.8 


4.0 


23.5 


4 


22.3 


1.03 


0 


0 


.3 


1.3 


2.0 


9.0 


16.7 


74.9 


3.3 


14.8 


6 


23.6 


.91 


0 


0 


0 


0 


4.0 


16.9 


17.6 


74.6 


2.0 


8.5 


3 


8.9 


.79 


0 


0 


0 


0 


2.3 


25.8 


6.2 


69.7 


.4 


4.5 


2 


15.2 


.70 


0 


0 


.5 


3.3 


4.7 


30.9 


8.8 


57.9 


1.2 


7.9 


1 


20.8 


.69 


0 


0 


0 


0 


7.2 


34.6 


12.8 


61.5 


.8 


3.9 



Category 5=Extemal Indicators: Community Development 

Category 4=Extemal Indicators: Career and Customized Education 

Category 6=Customer Satisfaction 

Category 3=General Education Indicators 

Category 2=Intemal Indicators 

Category l=Student Progression Indicators 



The perception of student services administrators about the use of effectiveness 
indicators leading toward positive change at their institutions ranges from a low mean 
score of .69 for Category 1 : Student Progression Indicators, to a high mean of 1 .06 for 
Category 5: External Indicators, Community Development. All effectiveness indicators 
were rated on the positive side from no change to positive change, and two indicators. 
Category 5: External Indicators, Community Development and Category 4: External 
Indicators, Career and Customized Education, rated between positive and very positive. 
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Table 19 displays the categories of indicators, in rank order of positive mean 
scores, for business affairs administrators and their perception about the use of 
effectiveness indicators leading to positive change at their community colleges. 



Table 19 

Perception of Use and Change: 
Business Affairs Administrators, 
Rank Order 



Cat. 


N 


Mean 


Very 

Negative 


Negative 


No Change 


Positive 


Very Positive 




22 




n 


% 


n 


% 


n 


% 


n 


% 


n 


% 




























5 


5.5 


1.18 5 


0 


0 


0 


0 


0 


0 


4.5 


81.8 


1.0 


18.2 


4 


8.3 


1.16 


0 


0 


0 


0 


.7 


8.4 


5.6 


67.5 


2.0 


24.1 


6 


5.2 


.94 


0 


0 


0 


0 


.8 


15.4 


3.6 


69.2 




15.4 


2 


3.7 


.87 


0 


0 


0 


0 


.75 


20.0 


2.75 


73.3 


.25 


6.7 


3 


2.0 


.85 


0 


0 


0 


0 


.4 


20.0 


1.5 


75.0 


.1 


5.0 


1 


6.8 


.65 


0 


0 


0 


0 


2.6 


38.2 


4.0 


58.8 


.20 


3.0 



Category 5=Extemal Indicators: Community Development 

Category 4=Extemal Indicators: Career and Customized Education 

Category 6=Customer Satisfaction 

Category 2=Intemal Indicators 

Category 3=General Education Indicators 

Category l=Student Progression Indicators 



The perception of business affairs administrators about the use of effectiveness 
indicators leading toward positive change at their institutions ranged from a low mean of 
.65 for Category 1 : Student Progression Indicators, to a high mean of 1. 18 for Category 
5: External Indicators, Community Development. All categories of effectiveness 
indicators were on the positive side of no change to positive change, with three categories 
of indicators above the rating of positive change. Positively rated indicators included: 
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Category 5: External Indicators, Community Development; Category 4: External 
Indicators, Career and Customized Education; and Category 6: Customer Satisfaction. 



93 



Table 20 

Perception That Use Led to Change: 

Comparison of Mean Scores by Administrative Position, 
Categorical Order 



Category 


Indicator 


All 


Presidents 


Instruction 


Student 

Services 


Business 

Affairs 






n=135 


n=25 


n=45 


n=43 


n-22 






11 


mean 


n 


mean 


n 


mean 


n 


mean 


n 


mean 


Category 1 


Student Progression 
Indicators 


56.4 


.79 


12.4 


.87 


19.2 


.88 


20.8 


.69 


6.8 


.65 


Category 2 


Internal Indicators 


45.0 


.79 


7.2 


.97 


18.7 


.90 


15.2 


.70 


3.7 


.87 


Category 3 


General Education 
Indicators 


29.4 


.97 


8.2 


1.04 


10.2 


1.08 


8.9 


.79 


2.0 


.85 


Category 4 


External Indicators: 
Career and Customized 
Education 


73.7 


1.15 


16.0 


1.29 


27.0 


1.17 


22.3 


1.03 


8.3 


1.16 


Category 5 


External Indicators: 

Community 

Development 


56.0 


1.20 


13.5 


1.26 


20.0 


1.30 


17.0 


1.06 


5.5 


U8 


Category 6 


Customer Satisfaction 


68.4 


.95 


16.6 


.92 


23.0 


1.01 


23.6 


.91 


5.2 


.94 



Note: Shaded areas indicate categories with the most positive mean scores by 
administrative position or the perception that the use led to positive change. 



Research Question 4 



Is there a difference by community college administrative position in the belief 
that they should use effectiveness indicators for initiating institutional change? 
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Summary of Significant Findings for Question 4 

First, all respondents believed that they should use effectiveness indicators in a 
very similar way. Second, all respondents believed Category 4: External Indicators, 
Career and Customized Education, and Category 6: Customer Satisfaction, would be the 
most valuable for initiating change at their community colleges. Third, presidents, 
instructional administrators and student services administrators believed Category 6: 
Customer Satisfaction, would be most valuable as a way to initiate change. Instructional 
administrators also believed Category 4: External Indicators, Career and Customized 
Education, would be most valuable as a way to initiate change. Fourth, business affairs 
administrators believed Category 4: External Indicators, Career and Customized 
Education would be most valuable to initiate change. Fifth, dispersion maps indicate a 
high level of agreement by each administrative subgroups in the belief that they should use 
effectiveness indicators to initiate change. Sixth, there is statistically no significant 
difference between administrative subgroups in the belief that they should use 
effectiveness indicators to initiate change. 

Presentation and Analysis of Data 

The following analysis examines the belief by all respondents and each 
administrative subgroup that they should use effectiveness indicators to initiate change at 
their institutions. The comparison is provided to examine the similarity and differences in 
the belief that they should use effectiveness indicators. Primary administrators’ belief that 
they should use effectiveness indicators to initiate institutional change was examined in 
research question 3 and the resulting mean scores were presented here in summary form. 

erJc 
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Table 21 

Perception of Indicator: 
Comparison of Mean Scores 
by Administrative Position, 
Categorical Order 



Category 


Indicator 


All 


Presidents 


Instruction 


Student 

Services 


Business 

Affairs 






n=135 


n=25 


n=45 


n=43 


n=22 
















Category 1 


Student Progression 
Indicators 


1.09 


; 1.02 


1.10 


1.11 


1.11 


Category 2 


Internal Indicators 


1.12 


1.11 


1.10 


1.14 


1.14 


Category 3 


General Education 
Indicators 


1.14 


1.09 


1.17 


1.15 


1.14 


Category 4 


External Indicators: 
Career and Customized 
Education 


1.03 


1.04 


i.o2 


1.04 


1.01 


Category 5 


External Indicators: 

Community 

Development 


1.14 


1.16 


1.13 


1.15 


1.11 


Category 6 


Customer Satisfaction 


1.03 


1:02 


1.02 


1.02 


1.10 



Note: Shaded areas indicate categories with the most positive mean score or scores. 



Table 21 depicts a comparison of the mean scores for all respondents and by each 
administrative position based on their response to the survey’s “Section A: Perception of 
the indicator: Do you believe this indicator is important or not important as a way to 
initiate change at your institution?” In the responses, the closer the mean value is to 1.0 
for each category the more affirmative were the responses. A mean score of 1.0 for a 
category would represent a completely affirmative response to the question. 

The responses by categorical order indicate general agreement in the belief that 
two categories should be used most to initiate change. All respondents, presidents, 
instructional administrators, and student services administrators all believed that Category 
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6: Customer Satisfaction would be most valuable. All respondents and instructional 
administrators believed equally that Category 4: External Indicators, Career and 
Customized Education, was also most valuable to initiate change. Business affairs 
administrators also rated Category 4: External Indicators, Career and Customized 
Education as their most valuable category. 

In general, the lowest rated categories varied between Category 3: General 
Education Indicators, Category 5: External Indicators, Community Development, and 
Category 2: Internal Indicators. The extent to which administrators’ believe they should 
use effectiveness indicators for initiating institutional changes was determined by the 
affirmative responses of primary community college administrators’ belief that they should 
use effectiveness indicators to initiate change within their respective institutions. The 
above analysis provided overall affirmative responses by all administrators in the belief that 
they should use effectiveness indicators to initiate change, ranging from 85.6% for General 
Education Indicators to 97.2% for External Indicators: Career and Customized Education. 

Dispersion of Each Administrative Group 

The data were further analyzed for dispersion or spread of variation within each 
subgroup on similarity of agreement about their belief that they should use indicators as a 
way to initiate change in their community colleges. The dispersion is shown by boxplots 
or stem and leaf plots: 

Boxplots are used to compare distributions. Each distribution is 
represented by a rectangular box whose length represents the variable’s inter- 
quartile range, on a scale corresponding to the observed values of the variable. A 
line, or ‘whisker,’ extends from each end of the box to the variable’s largest and 
smallest values, aside from those classed as outliers or extreme values. In SPSS, 
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an outlier is a value that is more than one and one-half box lengths from the end of 
the box, while an extreme value is a value that is more than three box lengths from 
the end of the box. (SPSS, 1997, p. 512). 



Figure 5 

Category One A: Student Progression Indicators, 
Perception of Indicator 
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Category 1 : Student Progression Indicators displays almost perfect agreement 
among community college presidents in their belief that they should use indicators to 
initiate change. The dispersion or spread of variation for instructional, student services 
and business affairs administrators displays a similar pattern and affirmative opinion about 
the use of indicators to initiate change. 



Figure 6 

Category Two A: Internal Indicators, 
Perception of Indicator 
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Category 2: Internal Indicators, displays a dispersion pattern similar for all four 
administrative sub groups. The opinion is also affirmative that they should use indicators 
to initiate change. 
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Figure 7 

Category Three A: General Education Indicators, 
Perception of Indicator 
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Category 3: General Education Indicators displays varying opinions and a number 
of outliers. In boxplots, outliers are values “ . . that are more than one and one-half box 
lengths from the end of the box/’ (SPSS, 1997, p. 512). Presidents in general have similar 
opinions although several outliers exist. Instructional and student services administrators 
vary but the spread of variation is less for instructional administrators, or they have similar 
opinions. Business affairs administrators displayed relatively close agreement about the 
value of using general education indicators for initiating change and they were similar to 
presidents in their opinions, although more outliers existed for business affairs 
administrators than for presidents. 
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Figure 8 

Category Four A: External Indicators, Career and Customized Education, 
Perception of Indicator 
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Category 4: External Indicators: Career and Customized Education displayed 
almost perfect agreement in all categories in the belief that they should use indicators to 
initiate change in their community colleges. The perception by all administrative 
subgroups was unanimously affirmative, with few outliers. 
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Figure 9 

Category Five A: External Indicators, Community Development, 
Perception of Indicator 
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instructional Admin. Business Affairs Adm 

Administrative Position 

Note: "Yes” equals 1 and "No" equals 2 

Category 5: External Indicators: Community Development displays a similar 
pattern of dispersion to Category 4: External Indicators, Career and Customized 
Education, with the exception of student services administrators who seem to display 
some spread in variation. 
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Figure 10 

Category Six A: Customer Satisfaction, 
Perception of Indicator 




Instructional Admin. Business Affairs Adm 



Administrative Position 

Note: "Yes" equals 1 and "No" equals 2 

Category 6: Customer Satisfaction displays for all administrative groups, with the 
exception of outliers, almost perfect affirmative belief that they should use this 
effectiveness indicator category for initiating change at their community colleges. 

Null Hypothesis 

Question four was further analyzed by testing the null hypothesis: There is no 
significant difference between community college presidents, instructional administrators, 
student services administrators and business affairs administrators in the belief that they 
should use effectiveness indicators as a way to initiate change in their institutions. The 
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question was analyzed using the one-way analysis of variance, ANOVA, with a fixed 
design to determine if any significant difference existed between the mean scores of each 
administrative position. The extent to which they differ was analyzed using each 
dependent variable “should use” by each effectiveness category and the independent 
variable of each administrative position. 

The ANOVA test for statistical significance found no statistically significant 
difference between the mean scores of community college presidents, instructional 
administrators, student services administrators and business affairs administrators in their 
belief that they should use effectiveness indicators to initiate change in all six categories of 
effectiveness indicators. The post hoc Boneferroni multiple comparisons as a result 
showed no group means as being statistically different. 

Therefore, the null hypothesis cannot be rejected or there is no statistically 
significant difference in the belief that they should use effectiveness indicators between 
community college presidents, instructional administrators, student services 
administrators and business affairs administrators. 





104 



Research Question 5 

Is there a difference by community college administrative position in the use of 
effectiveness indicators for initiating institutional change. 

Summary of Significant Findings for Question 5 

First, two categories of effectiveness indicators dominate the use by administrative 
groups. Category 4: External Indicators, Career and Customized Education and Category 
6: Customer Satisfaction. Second, instructional and business affairs administrators 
indicated they most used Category 4: External Indicators, Career and Customized 
Education. Third, presidents and student services administrators indicated they most used 
Category 6: Customer Satisfaction. Fourth, the categories where effectiveness indicators 
were most used for decision making included, in rank order by most use, planning, 
improving services, budgeting, accreditation, and program review. Fifth, administrative 
subgroups used effectiveness indicators most for decision making processes involving 
planning, although improving services and budgeting were also ranked first in at least one 
subgroup. Sixth, overall effectiveness indicators were used for decisions regarding 
legislative and community accountability as 9 th and 1 1 th respectively which was very low 
use in the list of decision making processes. Seventh, dispersions or spread of variations 
in the use of indicators was much wider than administrators belief that they should use 
indicators. This would indicate varying use by administrators. Eighth, business affairs 
administrators displayed about the same dispersion but lower use than other administrative 
subgroups. Ninth, ANOVA test found significant statistical differences between 
administrative subgroups. Tenth, post hoc multiple comparisons test showed a significant 
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difference between business affairs administrators and one or more other administrative 
subgroups in each category of indicators. 



Presentation and Analysis of Data 

Table 22 displays in categorical order, the mean scores for use of effectiveness 
indicators by each administrative position. 



Table 22 

Use of Indicators: 
Comparison of Mean Scores 
by Administrative Position, 
Categorical Order 



Category 


Indicator 


All 


Presidents 


Instruction 


Student 

Services 


Business 

Affairs 






Mean 


Mean 


Mean 


Mean 


Mean 
















Category 1 


Student Progression 
Indicators 


1.51 


1.42 


1.46 


1.47 


1.77 


Category 2 


Internal Indicators 


1.58 


1.50 


1.48 


1.60 


1.79 


Category 3 


General Education 
Indicators 


1.75 


1.64 


1.73 


1.75 


1.90 


Category 4 


External Indicators: 
Career and Customized 
Education 


1.40 


1.31 


1,30 


1.45 


1.59 


Category 5 


External Indicators: 

Community 

Development 


1.49 


1.37 


1.42 


1.52 


1.72 


Category 6 


Customer Satisfaction 


1.41 


1.28 


1.38 


1.36 


1.72 



Note: Shaded areas indicate mean values for categories with the highest administrative 
use. 



The lower the value of the mean score the more affirmative the response for 
having used the category of indicator. 
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The table depicts two primary categories which dominate the use of indicators for 
initiating change. The first category is Category 5: External Indicators, Career and 
Customized Education, most used by all respondents and individually by instructional and 
business affairs administrators. This category was also the second category most used by 
presidents and student services administrators. Indicators included in this category are 
placement rates in the workforce, employer assessment of students, and client/customer 
assessment of programs and services. The second category most used was Category 6: 
Customer Satisfaction. It was most used by presidents and student services 
administrators. This same category was the second most used category by the other 
administrative positions and was second most used by all respondents. 

Although there is general agreement which categories of indicators are most used 
by all administrative positions the extent in the use of effectiveness indicators appears to 
be different for each administrative position. The average or mean values display a pattern 
which indicates presidents use indicators most, followed by instructional administrators, 
student services administrators and business affairs administrators. The mean value of the 
business affairs administrators was seventeen points lower than the next closest score and 
twenty-five points below presidents. This indicates that the extent of use varies 
considerably between some positions. 

Use and Decision Making 

The difference in the use of effectiveness indicators can be further analyzed by 
examining the responses to question 1 6 which asked, “Have you in your area of 
responsibility used assessment information as input to the decision making process in any 
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of the following areas ? ” The areas included planning, budgeting, program review, 
accreditation, quality improvement measures, improving services, improving learning, 
improving occupational programs, improving general education outcomes, legislative 
accountability, community accountability and other as open ended. 

The responses to the use of indicators and/or assessment information for decision 
making processes is displayed in the following tables. Overall, planning, improving 
services, budgeting, accreditation and program review were rated as the most used. The 
ratings as portrayed in Table 23 range from a use of 77.3% to a high of 94.7%. The 
decision making process where they appeared to be least used was for community 
accountability. 



Table 23 

Use of Indicators in 
Decision Making Processes: 
All Respondents, 
Categorical Order 



Decision making process 


Rank 

Order 


Yes 

# % 


No 

# % 


Planning 


1 ■ 


126 


94.7 


7 


5.3 


Budgeting 




112 


84.2 


21 


15.8 


Program review 


5 


102 


77.3 


30 


22,7 


Accreditation 


4 


107 


81.1 


25 


18.9 


Quality improvement measures 


6 


81 


61.4 


51 


38.6 


Improving services 


2 


113 


85.6 


19 


14.4 


Improving learning 


7 


72 


54.5 


60 


45.5 


Improving occupational programs 


8 


69 


52.3 


63 


47.7 


Improving general education outcomes 


10 


59 


44.7 


73 


55.3 


Legislative accountability 


9 


66 


50.0 


66 


50.0 


Community accountability 


11 


58 


43.9 


74 


56.1 


Other 


12 


7 


5.3 


125 


94.7 



Note: Shaded areas indicate categories ranked in the top five by all respondents. 
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Presidents, shown in Table 24, used indicators most for planning, accreditation, 
budgeting, improving services, program review and improving occupational programs. 
Effectiveness indicators were least used for decision making processes involving 
community accountability. 



Table 24 

Use of Indicators in 
Decision Making Processes: 
Presidents, 

Categorical Order 



Decision making process 


Rank 

Order 


Yes 

# % 


No 

# % 


Planning 


1 


23 


95.8 


1 


4.2 


Budgeting 


' 2 


22 


91.7 1 


2 


8.3 


Program review 


■■■4 : 


20 


83.3 


4 


16.7 


Accreditation 


1 


23 


95.8 


1 


4.2 


Quality improvement measures 


6 


16 


66.7 


8 


33.3 


Improving services 


. ..3. ; 


21 


87.5 


3 


12.5 


Improving learning 


6 


16 


66.7 


8 


33.3 


Improving occupational programs 


: ■ -5 


18 


75.0 


6 


25.0 


Improving general education outcomes 


8 


14 


58.3 


10 


41.7 


Legislative accountability 


7 


15 


62.5 


9 


37.5 


Community accountability 


8 


14 


58.3 


10 


41.7 


Other 


9 


0 


00 


24 


100 



* Those with same percentages are rank ordered with the same number. 



Note: Shaded areas indicate categories ranked in the top five by presidents. 



Instructional administrators, shown in Table 25, used indicators most for decisions 
involving planning, program review, accreditation, improving services, improving 



occupational programs and budgeting. Effectiveness indicators again were least used for 
decision making processes involving community accountability. 
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Table 25 

Use of Indicators in 
Decision Making Processes: 
Instructional Administrators, 
Categorical Order 



Decision making process 


Rank 

Order 


Yes 

# % 


No 

# % 


Planning 


l 


41 


91.1 


4 


8.9 


Budgeting 


, ■ 5 ' 


36 


80.0 


9 


20.0 


Program review 


2 


40 


88.9 


5 


11.1 


Accreditation 


! ' T': 


38 


84.4 


7 


15.6 


Quality improvement measures 


7 


27 


60.0 


18 


40.0 


Improving services 


4 


37 


82.2 


8 


17.8 


Improving learning 




38 


84.4 


7 


15.6 


Improving occupational programs 


' 4 : 


37 


82.2 


8 


17.8 


Improving general education outcomes 


6 


30 


66.7 


15 


33.3 


Legislative accountability 


8 


22 


48.9 


23 


51.1 


Community accountability 


9 


21 


46.7 


24 


53.3 


Other 


10 


4 


8.9 


41 


91.1 



* Those with same percentages are rank ordered with the same number. 



Note: Shaded areas indicate categories ranked in the top five by instructional 
administrators. 



Student services administrators, shown in Table 26, used indicators most for 
decisions involving improved services, planning, program review, accreditation and 
budgeting. Unlike presidents and instructional administrators, they used indicators for 
decisions involving community accountability but used indicators least for improving 
general education outcomes. 
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Table 26 

Use of Indicators in 
Decision Making Processes: 
Student Services Administrators, 
Categorical Order 



Decision making process 


Rank 

Order 


Yes 

# % 


No 

# % 


Planning 


2 


41 


95.3 


2 


4.7 


Budgeting 




33 


76.7 


10 


23.3 


Program review 


,3 •. 


35 


81.4 


8 


18.6 


Accreditation 


'' a '""'' 


34 


79.1 


9 


20.9 


Quality improvement measures 


6 


26 


60.5 


17 


39.5 


Improving services 




42 


97.7 


1 


2.3 


Improving learning 


8 


17 


39.5 


26 


60.5 


Improving occupational programs 


10 


12 


27.9 


31 


72.1 


Improving general education outcomes 


11 


11 


25.6 


32 


74.4 


Legislative accountability 


7 


18 


41.9 


25 


58.1 


Community accountability 


9 


15 


34.9 


28 


65.1 


Other 


12 


2 


4.7 


41 


95.3 



Note: Shaded areas indicate categories ranked in the top five by student services 
administrators. 



Business affairs administrators, shown in Table 27, varied somewhat from other 
administrators in their use of indicators for decision making. They also used indicators for 
decisions involving planning, budgeting, improving services, accreditation and quality 
improvement measures, but unlike any of the other administrative positions, they used 
effectiveness indicators for decisions related to legislative accountability and community 



accountability. 
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Table 27 

Use of Indicators in 
Decision Making Processes: 
Business Affairs Administrators, 
Categorical Order 



Decision making process 


Rank 

Order 


Yes 

# % 


No 

# % 


Planning 




21 


100 


0 


0 


Budgeting 


■ : 1 


21 


100 


0 


0 


Program review 


6 


7 


35.0 


- 13 .. 


65.0 


Accreditation 


$• :: 


12 


60.0 


8 


40.0 


Quality improvement measures 


' 3 : 


12 


60.0 


8 


40.0 


Improving services 


: 2 


13 


65.0 


7 


35.0 


Improving learning 


9 


1 


5.0 


19 


95.0 


Improving occupational programs 


8 


2 


10.0 


18 


90.0 


Improving general education outcomes 


7 


4 


20.0 


16 


80.0 


Legislative accountability 


: 4 ■ : : : 


11 


55.0 


9 


45.0 


Community accountability 


;• 5 -:• 


8 


40.0 


12 


60.0 


Other 


10 


1 


5.0 


19 


95.0 



* Those with same percentages are rank ordered with the same number. 



Note: Shaded areas indicate categories ranked in the top five by business affairs 
administrators. 



Dispersion of Each Administrative Group 

The data was visually analyzed for its dispersion or spread of variation within each 
administrative group through the use of boxplots or stem and leaf plots. Boxplots visually 
display each category and its spread or variation for each subgroup or administrative 
position. 

Boxplots or stem and leaf plots display data in the following manner. ‘The solid 
black line is the median; the gray rectangle extends from the 25 th to the 75 th percentile 
(that is the middle 50% of the distribution; and the whiskers extend to the largest and 
smallest observed values within l'A times the height of each rectangle. If there are any 
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points outside the whiskers, they are labeled with case ID’s . . . and generally called 
outliers” (SPSS, 1996, p. 20). 
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Figure 1 1 

Category One B: Student Progression Indicators, 
Use of Indicator 




Instructional Admin. Business Affairs Adm 

Administrative Position 

Note: "Yes" equals 1 and "No" equals 2 



Category 1: Student Progression Indicators, shown in Figure 11, indicates much 
greater agreement between presidents than other groups. The dispersion and range is 
similar for instruction and student services administrators and quite different for business 
affairs administrators. The dispersion for business affairs administrators indicates a high 
non use of indicators and a fairly wide dispersion. 
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Figure 12 

Category Two B: Internal Indicators, 
Use of Indicator 
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Category 2: Internal Indicators, shown in Figure 12, displays similar dispersions 
for presidents and instructional administrators and less use of indicators by student service 
administrators and business affairs administrators. The dispersion for business affairs 
administrators indicates a fairly close agreement and low use of the indicator. 
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Figure 13 

Category Three B: General Education Indicators, 
Use of Indicator 
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Instructional Admin. Business Affairs Adm 

Administrative Position 

Note: "Yes” equals 1 and "No” equals 2 



Category 3: General Education Indicators, shown in Figure 13, indicates low use 
by all subgroups and varying dispersion or agreement within each subgroup. Presidents 
and instructional administrators have similar use rates and wide dispersion. Student 
services administrators have lower use but similar dispersion. Business affairs 
administrators have narrow dispersion and low use. 
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Figure 14 

Category Four B: External Indicators, Career and Customized Education, 
Use of Indicator 
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Instructional Admin. Business Affairs Adm 



Administrative Position 

Note: "Yes" equals 1 and "No" equals 2 



Category 4: External Indicators: Career and Customized Education, shown in 
Figure 14, depicts a variation in the dispersions. Presidents have a very broad dispersion 
of their use although it is fairly affirmative. Instructional administrators are similar to 
presidents. Student services administrators have narrower dispersion and display lower 
use. Business affairs administrators have a fairly broad dispersion and low use which is 
quite different from the other three subgroups. 
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Figure 15 

Category Five B: External Indicators, Community Development 
Use of Indicator 
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Administrative Position 

Note: "Yes" equals 1 and "No” equals 2 



Category 5: External Indicators, Community Development, shown in Figure 15, 
displays dispersions for presidents and instructional administrators which indicate broad 
dispersion but high use. Student services administrators display a very wide dispersion or 
use of the indicators but reflect fairly high use. Business affairs administrators show a 
dispersion which is narrower and the lowest use of the subgroups. 
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Figure 16 

Category Six B: Customer Satisfaction, 
Use of Indicator 
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Category 6: Customer Satisfaction, shown in Figure 16, displays a high use for 
presidents, instructional administrators and student services administrators but the 
dispersion of presidents is narrower than the other two subgroups. Business affairs 
administrators have a dispersion similar to presidents but inversely displays non use of the 
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Null Hypothesis 

Question five was further analyzed beyond normal frequency responses by testing 
the null hypothesis: There is no significant difference between presidents, instructional 
administrators, student services administrators and business affairs administrators in the 
use of effectiveness indicators for initiating institutional change. The question was 
examined using the one-way analysis of variance, ANOVA, with a fixed design to 
determine if a significant difference existed between the mean scores of each 
administrative position. The extent to which they differ was analyzed by using the 
dependent variable “are using” by each effectiveness category and the independent variable 
of each administrative position. 

The ANOVA test for statistical significance found a significant difference between 
the mean scores of community college presidents, instructional administrators, student 
services administrators and business affairs administrators at the .05 level 
of significance. Therefore, we reject the null hypothesis of no significant difference 
between the mean scores of each administrative position. The post hoc Boneferroni 
multiple comparisons test showed significant differences by administrative position in the 
use of effectiveness indicators in the following categories. 

In category IB, Student Progression Indicators, statistically significant differences 
were found between business affairs administrators, community college presidents, 
instructional and student services administrators at the .006, .001 and .002 levels of 
significance respectively. The basis for this could be that the effectiveness indicators used 
in the survey were more related in application to the other three positions and less to 
business affairs administrators. There may also be a lack of recognition about their use in 
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major decisions as indicated by business affairs administrators in question 16 which 
surveyed how they were used in decision making processes. 

Category 2B, Internal Indicators, statistically significant differences were found by 
administrative position between business affairs administrators, community college 
presidents and instructional administrators at the .020 and .001 level of significance 
respectively. There was however, no significant difference found between business affairs 
administrators and student services administrators. The same reasoning could be applied 
here as suggested in category one above. 

Category 3B, General Education Indicators, a statistically significant difference by 
administrative position was found between business affairs administrators and community 
college presidents at the .023 level of significance but not with either instructional 
administrators or student services administrators. In Category 3B, indicators were used 
which relate directly to general education and may appear to be more directly related to 
learning outcomes and appear to have little relationship to business affairs administrators 
in the normal operation of a community college. This could lead to the difference of 
significance between the two positions. 

Category 4B, External Indicators: Career and Customized Education, statistically 
significant differences were found between business affairs administrators, community 
college presidents and instructional administrators. The use of indicators in this category 
may again reflect the role of the administrator within the institution since these more 
closely reflect the role of the president and instructional administrators. The indicator is 
more directly related to the development and delivery of career and customized education. 
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a role more common to instruction and college presidents, and less to business affairs 
administrators given normal division of functions. 

Category 5B, External Indicators: Community Development, statistically 
significant differences were found between business affairs administrators and community 
college presidents. The significant difference here in the use of indicators may relate to 
their routine functions since these may be more related to the external role of a 
community college president and less to the normal functioning of business affairs 
administrators. 

Category 6B, Customer Satisfaction, statistically significant differences were found 
between business affairs administrators, community college presidents, instructional 
administrators and student services administrators at the .000, .001 and .000 levels of 
significance respectfully. The statistically significant differences in the means can again 
probably be attributed to the various roles performed by each administrative position and 
the use of the indicators which might be perceived to be more related to the other 
positions. There appears, however, to be an inability by business affairs administrators to 
apply the use of the various indicators, as suggested by accreditation, to decision making 
processes involving planning, budgeting or other institutional improvement decisions. 
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Research Question 6 

Is there a difference by community college administrative position in the belief that the 
use of effectiveness indicators led to positive changes in their institutions. 

Summary of Significant Findings For Question 6 

First, Category 5: External Indicators, Community Development, dominated the 
perception by administrative subgroups that its use led to positive change in community 
colleges. Second, Category 4: External Indicators, Career and Customized Education, 
was also rated first by presidents. Third, the dispersion of variances in each 
administrative subgroup was similar within each category but varied somewhat between 
categories. Fourth, the perception of each administrative subgroup was positive to very 
positive about the use of indicators leading to positive change in each category of 
indicators. Fifth, no statistically significant differences were found between administrative 
subgroups. Therefore, the null hypothesis could not be rejected. 

Presentation and Analysis of Data 

The extent to which effectiveness indicators led to positive change in community 
colleges was measured by the number of positive responses to the Likert type scale by 
community college administrators where they had already indicated their use of indicators 
as a means of initiating change at their institutions. The survey questioned whether they 
believed that the use of effectiveness indicators led to positive change in their respective 
community colleges. The question was analyzed by examining the difference in 
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administrators’ belief that the use of effectiveness indicators led to change at their 
institutions. 



Table 28 

Perception That Use Led to Change: 

Comparison of Mean Scores by Administrative Position, 
Categorical Order 



Category 


Indicator 


All 


Presidents 


Instruction 


Student 

Services 


Business 

Affairs 






n=l 


135 


n=25 


n=45 


n=43 


n=22 






n 


mean 


n 


mean 


n 


mean 


n 


mean 


n 


mean 


Category 1 


Student Progression 
Indicators 


56.4 


.79 


12.4 


.87 


19.2 


.88 


20.8 


.69 


6.8 


.65 


Category 2 


Internal Indicators 


45.0 


.79 


7.2 


.97 


18.7 


.90 


15.2 


.70 


3.7 


.87 


Category 3 


General Education 
Indicators 


29.4 


.97 


8.2 


1.04 


10.2 


1.08 


8.9 


.79 


2.0 


.85 


Category 4 


External Indicators: 
Career and Customized 
Education 


73.7 


1.15 


16.0 


1.29 


27.0 


1.17 


22.3 


1.03 


8.3 


1.16 


Category 5 


External Indicators: 

Community 

Development 


56.0 


1.20 


13.5 


1.26 


20.0 


1.30 ; 


17.0 


1-06 


5.5 


; 1-18 


Category 6 


Customer Satisfaction 


68.4 


.95 


16.6 


.92 


23.0 


1.01 


23.6 


.91 


5.2 


.94 



Note: Shaded areas indicate categories with the most positive mean scores by 
administrative position or the perception that the use led to positive change. 



The category, shown in Table 28, rated by all respondents as most positive in 
leading to positive change at their institutions, with a mean score of 1 .20, was Category 5, 
External Indicators: Community Development. This category includes indicators on 
responsiveness to the community and participation rate in the service area. This same 
category was rated highest by instructional administrators with a mean score of 1.30 
followed by business affairs administrators with a mean score of 1 . 1 8 and student services 
administrators with a mean score of 1.06. Presidents rated Category 5, External 
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Indicators: Community Development their second highest mean score, and second overall 
only to instructional administrators. Presidents believed, with a mean score of 1 .29, 
Category 4, External Indicators: Career and Customized Education led to the most 
positive change. 

Two categories of indicators believed by all respondents to be least useful in 
leading to institutional change, with a mean of .79, was Category 1, Student Progression 
Indicators, which includes student goal attainment, student persistence (Fall to Fall) and 
program completion. The second category was Category 2, Internal Indicators, with a 
mean score of .79 and includes the number and rate who transfer, performance after 
transfer and success of remedial or developmental students in subsequent related college 
level course work. 

Dispersion of Each Administrative Group 

The question was further analyzed using boxplots or stem and leaf dispersion maps 
to display the spread of variation in their belief by administrators that the use of 
effectiveness indicators led to positive change at their institutions. 
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Figure 17 

Category One C: Student Progression Indicators, 
Use and Perception of Change 
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Instructional Admin. 
Administrative Position 



Business Affairs Adm 



Category 1: Student Progression Indicators, shown in Figure 17, displays data for 
presidents and instructional administrators which are similar. Both depict positive 
attributes about the use of indicators and fairly close agreement within the subgroups with 
the exception of outliers. Student services and business affairs administrators depict wide 
spreads of variation. They are internally less in agreement and their perception is less 
positive than either presidents or instructional administrators. 
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Figure 18 

Category Two C: Internal Indicators, 
Use and Perception of Change 
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Instructional Admin. 
Administrative Position 



Business Affairs Adm 



Category 2: Internal Indicators, shown in Figure 18, displays fairly close 
agreement among college presidents about the positive attributes of having used these 
indicators. Instructional administrators display more spread and wider range of variation 
but have a fairly positive opinion slightly below presidents. Student service administrators 
show a similar pattern but slightly narrower range of dispersion, which is less positive than 
instructional administrators. Business affairs administrators, although somewhat similar in 
dispersion on the positive side, show and display a more positive perception about having 
used the indicator. 
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Figure 19 

Category Three C: General Education Indicators, 
Use and Perception of Change 
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Instructional Admin. Business Affairs Adm 

Administrative Position 



Category 3: General Education Indicators, shown in Figure 19, display fairly close 
agreement in several subgroup categories. Presidents display narrow dispersion or 
agreement about the value of having used this indicator. Instructional administrators 
display very close agreement about the value of having used indicators but have a number 
of outliers. Student services administrators show the widest dispersion or least agreement 
in their subgroup about the value of having used this indicator. Although only a few 
business affairs administrators have used the indicator, they agree their use was positive. 
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Figure 20 

Category Four C: External Indicators, Career and Customized Education, 
Use and Perception of Change 




Instructional Admin. Business Affairs Adm 

Administrative Position 



Category 4: External Indicators: Career and Customized Education, shown in 
Figure 20, displays a very broad dispersion by presidents but is very positive at the same 
time. Instructional and business affairs administrators have narrow variation or display 
similar opinions about the value of having used this indicator although they do have a 
number of outliers. Student services administrators are slightly less in agreement about 
the value of using indicators and have the lowest positive opinion about having used this 
indicator. 
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Figure 21 

Category Five C: External Indicators, Community Development, 
Use and Perception of Change 
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Category 5: External Indicators: Community Development, shown in Figure 21, 
indicates similar positive opinions for each subgroup. They display, however, quite 
different dispersion of variances. Presidents display a very positive opinion about having 
used indicators with some variation to the positive side. Instructional administrators 
display a very positive opinion with a wide dispersion towards those positive opinions. 
Student services administrators display narrow variation but the least positive opinion. 
Business affairs administrators, although few in number, agree very strongly about the 
value of having used effectiveness indicators which led to positive change. 
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Figure 22 

Category Six C: Customer Satisfaction, 
Use and Perception of Change 
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Category 6: Customer Satisfaction, shown in Figure 22, displayed positive 
opinions by all administrative subgroups with similar narrow dispersions or spreads in 
variation. A number of outliers are shown for each subgroup. 

Null Hypothesis 

This still leaves somewhat unanswered the question “Is there a difference , by 
administrative position , in the belief that the use of effectiveness indicators led to positive 
change in their institution ?” The question was tested using the null hypothesis: There is 



er|c 



14 4 



130 

no significant difference between community college presidents, instructional 
administrators, student services administrators and business affairs administrators in the 
belief that the use of effectiveness indicators led to institutional change. The question was 
analyzed by using a one-way analysis of variance, ANOVA, to determine if significant 
differences existed between the mean scores of each administrative position. “A 
statistically significant test tells you only that it is unlikely that all the group means in the 
population are different, but not which groups are different from each other. The multiple 
comparisons test will give us this information” (SPSS, 1996, p. 138). The extent to 
which they differ was analyzed by using the dependent variable “perception of use and 
change” by each effectiveness category and the independent variable of each 
administrative position. To determine which group means were different the post hoc 
Boneferroni multiple comparisons test was used and based on a .05 level of significance. 

In Question six, the ANOVA and post hoc Boneferroni multiple comparisons test 
showed no statistically significant difference between the mean scores of presidents, 
instructional administrators, student services administrators and business affairs 
administrators. Therefore, the null hypothesis can not be rejected or there is no significant 
difference in the belief that the use of effectiveness indicators led to institutional change 
between community college presidents, instructional administrators, student services 
administrators and business affairs administrators. 
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Chapter Summary 



Research Question 1 

Do community college administrators believe they should use effectiveness indicators for 
initiating institutional change? 

Findings indicate first, overwhelming support was given by administrators about 
the value of using effectiveness indicators to initiate change. Support ranged from 85.5% 
to 97.0%. Second, three categories of indicators were perceived above a 90.0% level by 
all respondents, to be important as a way to initiate change at their institutions. These 
categories included: Category 4: External Indicators, Career and Customized Education, 
Category 6: Customer Satisfaction, and Category 1: Student Progression Indicators. 

Third, presidents rated highest the value of using Category 6: Customer Satisfaction, at 
98.3%. Fourth, instructional administrators rated highest the value of using Category 4: 
External Indicators, Career and Customized Education, at 97.8%. Fifth, student services 
administrators rated highest the value of using Category 6: Customer Satisfaction, the 
same as community college presidents, at 98. 1%. Sixth, business affairs administrators 
rated highest the value of using Category 4: External Indicators, Career and Customized 
Education, at 98.6%. This was also the highest rating for any single category by an 
administrative group. 

Research Question 2 

Since the last community college regional accreditation (1990 or later) have 
community college administrators used effectiveness indicators for initiating change and 
improvement? 
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Findings indicate first, all respondents rated four categories of indicators above the 
50% level of usage. These categories included: Category 4: External Indicators, Career 
and Customized Education; Category 6: Customer Satisfaction; Category 5: External 
Indicators, Community Development; and Category 1: Student Progression Indicators. 
Second, presidents indicated they used Category 6: Customer Satisfaction, the most at 
72.0%, as a way to initiate change. Third, instructional administrators indicated they used 
Category 4: External Indicators, Career and Customized Education, the most at 69.7%, as 
a way to initiate change. Fourth, student services administrators indicated they used 
Category 6: Customer Satisfaction Indicators, the most at 63.9%, as a way to initiate 
change. Fifth, business affairs administrators indicated they used Category 4: External 
Indicators, Career and Customized Education Indicators, the most at 40.9%, as a way to 
initiate change. Sixth, the least used category was Category 3: General Education 
Indicators at 30.6%. Seventh, indicators were used most for input to decision making 
processes involving planning, improving services, budgeting, accreditation, and program 
review. Eighth, indicators were used at low levels for decisions regarding improved 
learning, ranked number seven, and community accountability ranked number eleven. 

Research Question 3 

Do community college administrators believe that the use of effectiveness indicators led 
to positive changes or improvements in their institutions ? 

The findings indicate first, all respondents believed that Category 5: External 
Indicators, Community Development, was the indicator which led to the most positive 
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change. Second, presidents believed Category 4: External Indicators, Career and 
Customized Education, was the indicator which led to the most positive change. Third, 
instructional, student services and business affairs administrators believed Category 5: 
External Indicators, Community Development, was the indicator which led to the most 
positive change. Fourth, all respondents rated two categories between positive and very 
positive. The two categories were Category 5: External Indicators, Community 
Development and Category 4: External Indicators, Career and Customized Education, as 
the indicators which led to the most positive change. 

Research Question 4 

Is there a difference by community college administrative position in the belief 
that they should use effectiveness indicators for initiating institutional change? 

The findings indicate first, all respondents believed that they should use 
effectiveness indicators in a very similar way. Second, all respondents believed Category 
4: External Indicators, Career and Customized Education, and Category 6: Customer 
Satisfaction, would be the most valuable for initiating change at their community colleges. 
Third, presidents, instructional administrators and student services administrators believed 
Category 6: Customer Satisfaction, would be most valuable as a way to initiate change. 
Instructional administrators also believed Category 4: External Indicators, Career and 
Customized Education, would be most valuable as a way to initiate change. Fourth, 
business affairs administrators believed Category 4: External Indicators, Career and 
Customized Education would be most valuable to initiate change. Fifth, dispersion maps 
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indicate a high level of agreement by each administrative subgroups in the belief that they 
should use effectiveness indicators to initiate change. Sixth, there is statistically no 
significant difference between administrative subgroups in the belief that they should use 
effectiveness indicators to initiate change. 

Research Question 5 

Is there a difference by community college administrative position in the use of 
effectiveness indicators for initiating institutional change? 



The findings indicate first, two categories of effectiveness indicators dominate the 
use by administrative groups. Category 4: External Indicators, Career and Customized 
Education and Category 6: Customer Satisfaction. Second, instructional and business 
affairs administrators indicated they most used Category 4: External Indicators, Career 
and Customized Education. Third, presidents and student services administrators 
indicated they most used Category 6: Customer Satisfaction. Fourth, the categories where 
effectiveness indicators were most used for decision making included, in rank order by 
most use, planning, improving services, budgeting, accreditation, and program review. 
Fifth, administrative subgroups used effectiveness indicators most for decision making 
processes including planning, although improving services and budgeting were also ranked 
first in at least one subgroup. Sixth, overall effectiveness indicators were used for 
decisions regarding legislative and community accountability as 9 th and 1 1 th respectively 
which was very low use in the list of decision making processes. Seventh, dispersions or 
spread of variations in the use of indicators was much wider than administrators’ belief 
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that they should use indicators. This would indicate varying use by administrators. 

Eighth, business affairs administrators displayed about the same dispersion but lower use 
than other administrative subgroups. Ninth, ANOVA test found significant statistical 
differences between administrative subgroups. Tenth, post hoc multiple comparisons test 
showed a significant difference between business affairs administrators and one or more 
other administrative subgroups in each category of indicators. 

Research Question 6 

Is there a difference by community college administrative position in the belief that the 
use of effectiveness indicators led to positive changes in their institutions ? 

The findings indicate first. Category 5: External Indicators, Community 
Development, dominated the perception by administrative subgroups that its use led to 
positive change in community colleges. Second, Category 4: External Indicators, Career 
and Customized Education, was also rated first by presidents. Third, the dispersion of 
variances in each administrative subgroup was similar within each category but varied 
somewhat between categories. Fourth, the perception of each administrative subgroup 
was positive to very positive about the use of indicators leading to positive change in each 
category of indicators. Fifth, no statistically significant differences were found between 
administrative subgroups. Therefore, the null hypothesis could not be rejected. 
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Chapter V : Summary of Findings 
Conclusions, Implications and Recommendations 

The purpose of this study was to examine in selected northwest community 
colleges that have undergone accreditation since 1990, the perceived importance of using 
effectiveness indicators for assessing institutional outcomes and investigate their 
congruence with actual or potential use for making institutional changes and 
improvements. 

General objectives of the study included: 

1 . To determine which effectiveness indicators are suggested by the accreditation 
process and related literature and can be used for assessing institutional 
effectiveness; 

2. To determine what the research literature has to say about the use of 
effectiveness indicators in assessment practice and for use in initiating 
institutional change. 

Research objectives of the study included: 

1. To determine if community college administrators believe they should use 
effectiveness indicators for initiating institutional change; 

2. To determine if community college administrators whose institutions have 
been accredited since 1990, have used effectiveness indicators to initiate 
change in their respective community colleges; 

3. To determine if community college administrators believe that the use of 
effectiveness indicators led to positive change at their institutions; 
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4 . To determine if there was a difference by community college administrative 
position in the belief that they should use effectiveness indicators for initiating 
institutional change; 

5 . To determine if there was a difference by community college administrative 
position in the use of effectiveness indicators for initiating institutional changes; 

6. To determine if there was a difference by community college administrative 
position in the belief that the use led to positive changes in their community 
colleges. 

In order to accomplish this task, eight questions guided this study. 

General questions that set the context and guided the literature review were: 

1 . What kind of purposely selected effectiveness indicators are suggested by the 
accreditation process and related literature, and can be used to base 
institutional effectiveness assessment? 

2. What does the research literature have to say about the use of effectiveness 
indicators in assessment practices and for making institutional change? 

Research questions that guided this investigative quantitative study were: 

1 . Do community college administrators believe they should use effectiveness 
indicators for initiating institutional change? 

2. Since the last community college regional accreditation (1990 or later) have 
community college administrators used effectiveness indicators for initiating 
institutional change? 

3. Do community college administrators believe that the use of effectiveness 
indicators led to positive changes in their institutions? 
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4. Is there a difference by community college administrative position in the 
perception about the perceived importance of using effectiveness indicators for 
initiating institutional change? 

5. Is there a difference by community college administrative position in the use of 
effectiveness indicators for initiating institutional change? 

6. Is there a difference by community college administrative position in the 
perception that the use of effectiveness indicators led to positive changes in the 
institution? 

Survey research was used to conduct a study of primary community college 
administrators serving in community colleges accredited since 1990 by the Northwest 
Association of Schools and Colleges. 

A 24 item questionnaire was mailed in May and June, 1997, to 232 potential 
respondents representing 59 northwest community colleges. After removing invalid 
responses, particularly those completed by someone other than the intended recipient, 135 
valid responses were used in the data analysis which included 7 duplicated for equal 
weight where the administrator represented both instruction and student services. The 
valid response rate was 55.2%. The return rate included: presidents at 25 or 42.4%; 
instructional administrators at 45 or 76.3%; student services administrators at 43 or 
72.9%; and business affairs administrators at 22 or 37.3%. 

The questionnaire was based on six categories of indicators representing 21 
different suggested measures. The indicators were primarily derived for the AACC Core 
Indicators Model except for Category 3: General Education and Category 6: Customer 
Satisfaction which were derived from the literature review and past institutional practice. 
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Descriptive analysis was used to examine research questions one, two and three. 
Primarily mean scores, distributions and rank orders were used to examine the belief that 
community college administrators should use effectiveness indicators, that they use 
effectiveness indicators, and that they believe the use led to positive change in their 
respective community colleges. The questions were further analyzed to determine which 
categories had the most and least support for use at initiating institutional change. 

Questions four, five and six examined the differences between primary 
administrators and their belief that they should use effectiveness indicators, have used 
effectiveness indicators and the belief that the use led to positive change. The differences 
were analyzed using descriptive statistics and boxplots to visually analyze dispersion or 
degree of variances within each category and group. 

The extent to which these subgroups of primary administrators differed in the 
belief or behavior was analyzed using an ANOVA to determine if there were statistically 
significant differences. A post hoc Boneferroni multiple comparisons test was used to 
determine that if there were statistically significant differences, which group means were 
different. 



Summary of Findings 

All administrators believe that they should use effectiveness indicators to initiate 
institutional change. Support ranged ffom 85.5% to 97.0%. Three categories of 
indicators were rated above the 90% level. They included Category 4: External 
Indicators, Career and Customized Education; Category 6: Customer Satisfaction; and 
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Category 1 : Student Progression Indicators. The lowest rating for a single category was 
Category 3: General Education Indicators. 

All administrators used effectiveness indicators at a much lower rate than they 
believed they should use them. Only four indicators were used by all administrators above 
the 50% level: Category 4: External Indicators, Career and Customized Education; 
Category 6: Customer Satisfaction; Category 5: External Indicators, Community 
Development; and Category 1 : Student Progression Indicators. The least used category 
by all respondents was Category 3: General Education Indicators. Administrators used 
effectiveness indicators most for decision making processes involving planning, improving 
services, budgeting, accreditation and program review. They used effectiveness indicators 
least for community accountability. 

Administrators believed that the use of Category 5: External Indicators, 
Community Development was the indicator that led to the most positive change in their 
institutions. Administrators believed that the use of two categories led to positive or very 
positive change: Category 5: External Indicators, Community Development; and Category 
4: External Indicators, Career and Customized Education. 

All respondents affirmatively agreed that they should use effectiveness indicators. 
Administrators’ beliefs varied between Category 6: Customer Satisfaction and Category 4: 
External Indicators, Career and Customized Education, as to which indicator would be 
most useful to initiate change. Administrators had a high level of agreement between each 
administrative subgroup, or low dispersion of variances in the belief that they should use 
effectiveness indicators. Administrative subgroups had no statistically significant 
difference in their belief that they should use effectiveness indicators to initiate change. 
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Administrators use two groups of effectiveness indicators most: Category 4: 
External Indicators, Career and Customized Education; and Category 6: Customer 
Satisfaction. Administrators used effectiveness indicators most for decision making 
processes involving planning, improving services, budgeting, accreditation and program 
review. Administrative subgroups ranked planning first and most frequently. Dispersion 
of spread of variation was much wider than belief in the use of effectiveness indicators, 
although use was moderate except for business affairs administrators. Business affairs 
administrators showed similar dispersion but low use of indicators. Significant statistical 
differences were found for administrative subgroups and post hoc multiple comparisons 
test showed significant differences between business affairs administrators and one or 
more other administrative subgroups. 

All administrators who have used effectiveness indicators believe that Category 5: 
External Indicators, Community Development led to the most positive change. Category 
4: External Indicators, Career and Customized Education was also rated most useful by 
presidents. The dispersion of variances between administrative subgroups for each 
category were similar but varied between categories. The belief that the use led to 
positive change was positive to very positive for each administrative subgroup in each 
category of indicators. No statistically significant differences could be found between 
administrative subgroups in the belief that the use of indicators led to positive change. 
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Conclusions 



Although it may seem self evident, almost all administrators believed they should 
be using effectiveness indicators to initiate change at their institutions. This perception 
check was important since without the belief in the value of using indicators, behavior or 
use of the indicators to initiate change probably would not take place. 

The study confirms the belief in the use of an institutional effectiveness model as 
required by the Northwest Association of Schools and Colleges, but indicates a lack of 
application and use of this model, to any major extent, in the normal institutional 
functioning and decision making process. The study did provide evidence that where it 
was used there were positive results from initiating institutional change and improvement. 

The result of community colleges not adopting meaningful and useful effectiveness 
indicators, publicly reported to our stakeholders and intentionally used for on-going 
change and improvement, leads to public policies requiring performance indicators in 
budget authorization bills or other legislative mandates as is currently prevalent in many 
states including the northwest. 

Educators believe in change and improvement focused on learners and learning. 
This belief is firmly held. We must however, through our actions, convince our publics of 
our continuous improvement and publicly report our successes in varying degrees and 
include those areas needing improvement and our anticipated action for improving weak 
performance. 
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Community colleges claim to be focused on the needs of their unique districts, but 
we must confirm our successes and tell our stories in meaningful and convincing ways. 
Effectiveness models using indicators must reflect each unique institutional mission and 
present a picture, based on qualitative and quantitative evidence, which is convincing to 
our stakeholders if we expect continued or enhanced public support. 

Effectiveness indicators must always allow us to reflect on learners and learning in 
meaningful ways. First, models of institutional effectiveness must begin at the core of the 
educational process and focus on the individual students as learners. Second, the model 
must focus on the department, division or programmatic level and provide quantitative 
and qualitative information used for change and improvement in programs. Third, 
effectiveness models must provide meaningful institutional effectiveness indicators which 
can be viewed externally by the public and not only tell our community college story but 
provide meaningful information for internal decision making and institutional change and 
improvement. 

The failure of using an effectiveness model to initiate change and improvement, as 
well as provide accountability to our communities and state stakeholders, will lead to 
publicly mandated performance or accountability measures which may or may not reflect 
our unique missions and at worst could reflect partisan politics. If educational institutions 
wish to remain independent to achieve their unique missions we must use an effectiveness 
model which provides us the tools for self improvement and the information necessary to 
tell our story to our publics. The institutional effectiveness model required by 
accreditation and embraced by community colleges can and must be used to achieve this 
goal. 
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Implications and Recommendations 

The Northwest Association of Schools and Colleges must enforce Standard 5, 
Policy #25, now known as Policy Statement 2.2, Policy on Educational Assessment, which 
requires the adoption of an institutional effectiveness model. Failure by community 
colleges to adopt this model should result in censure by the Northwest Association of 
Schools and Colleges until compliance is achieved. 

Recommendation 1 

Institutions must embrace the institutional effectiveness model as a mechanism 
useful for the continuous improvement process necessary for community college 
improvement and the achievement of student learning. 

Recommendation 2 

Faculty, staff, administrators and boards must be trained in the application and use 
of the institutional effectiveness model. There is an apparent lack of knowledge and 
understanding by faculty, staff, administrators and boards in using an effectiveness model. 
There is a need for on-going training in using an effectiveness model for day to day and 
year to year decision making and improvement practices. This training could be provided 
by the Northwest Association of Schools and Colleges, professional associations, or 
internally by knowledgeable educators. 
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Recommendation 3 

Boards must establish outcomes or institutional goals based on their unique 
community college mission and establish the foundation for the establishment of other 
institutional goals and objectives. 

Recommendation 4 

Boards must hold the community college president or CEO accountable for the 
achievement of these institutional outcomes. 

Recommendation 5 

Continuous assessment as a way of thinking and doing must be completed at all 
levels of the institution by using quantitative and qualitative assessment in the evaluation 
of outcomes or processes. 

Recommendation 6 

Institutions must adopt an assessment model prominently placed within the 
organization, responsible to the President, for the development and implementation of an 
assessment process with the involvement of all internal stakeholders. All assessments 
must be developed, applied, interpreted and used in conjunction with college programs 
and personnel. Decisions must reflect goals and assessment not status quo or political 
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Recommendation 7 

Indicators and measures must ultimately reflect the institutions’ goals and 
objectives. Measures must be developed at each level or process by the persons closest to 
the process, including faculty, staff and administrators. 

So What if We Don’t 

Accountability measures in the form of performance measures will be adopted by 
legislatures, state boards, or even the Office of Education at the federal level. When peer 
review fails to enforce its own accreditation processes, the public will ultimately react. 

The development of public policies, sometimes even showing up in budget bills, may 
reflect partisan politics more than good educational practices. Unsubstantiated public 
educational policies can lead to unfocused goals, objectives, and compliance behavior, 
which ultimately leads to the antithesis of both good educational practices and a focus on 
students as learners and the learning process we are trying to improve. This focus is 
usually best understood by educational professionals closest to the student. We must 
enforce our own standards of accreditation and embrace the institutional effectiveness 
model. 



Recommendations for Future Study 
Based on the findings and implications of this study, the following 
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1 . This study could be conducted on a much broader audience and include both 
Boards of Trustees and faculty to examine why there is low use of the 
effectiveness indicators. 

2. A study could be conducted to identify why there is an inability to use 
effectiveness indicators for routine functioning of a community college. 

3. A study could be conducted to identify the best practices used by community 
colleges who have used effectiveness indicators and believe that they led to 
positive change. 

4. A study could be conducted to identify detractors in the use of effectiveness 
indicators in the functioning of a community college. 
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X 


X 


X 


X 


X 


X 


Contextual 






X 








Return on Investment 


X 












Customer Satisfaction 


X 


X 




X 
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Matrix Table: Assessment Purpose and Indicators 





Harris 


Lutz 


S. Alabama 


Chu 


Dennis- 


Brown 








Skills Center 




Rounds 






1995 


1992 


1993 


1991 


1989 


1991 


(A) FACTS 














Theory or Model 


X 












Development 














Applied Research 


X 


X 


X 


X 


X 


X 


Single Institution 


X 












Multiple Institution 




X 


X 


X 


X 


X 
















(B) PURPOSE 














Compliance 






X 








Persuading 














Monitoring 








X 




X 


Improving 


X 


X 


X 






X 


Judging 




X 




X 




X 
















(C) TYPES OF 














INDICATORS USED 














Input 






X 


X 


X 




Process (Environmental) 


X 


X 


X 




X 


X 


Outcomes 


X 


X 


X 


X 


X 


X 


Contextual 


X 












Return on Investment 














Customer Satisfaction 


X 


X 
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Dear Colleague: 

Within the next few days you will receive a request to complete a brief questionnaire. 
This is a survey about assessment and its use to initiate change in community colleges. 
The survey is being conducted as part of my doctoral studies in Education, Community 
College Leadership, at Oregon State University and is co-sponsored by Highline 
Community College in Des Moines, Washington. 

Your immediate and reflective thoughts will add to the conversation, information and 
knowledge about the use of assessment information in Northwest Community Colleges. 
Thank you in advance for your assistance. 



Sincerely, 



Jim Sorensen 

Vice President for Students 
Highline Community College 
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Dear Colleagues: 

Institutional effectiveness in community colleges is vital to education. 

As a President, Chief Instructional, Chief Student Services, or Chief Business Affairs 
Officer in a northwest community college, I need to know your attitudes and behaviors 
about the use of assessment information and its application for initiating institutional 
change. 

You are part of a select number of persons from northwest community colleges who have 
completed a self study since 1990. Your response is vitally important. The questionnaires 
are coded for categorization and follow-up of non respondents, but your responses are 
completely confidential. 

Please return your questionnaire by June 15, 1997. If you have any questions, please call 
me at 206-878-3710 ext. 3503, or write to me at the enclosed address. 

Sincerely, 



Jim Sorensen 

Vice President for Students 
Highline Community College 
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Dear Colleague, 

Last week, you received a questionnaire seeking your opinions about the use of 
assessment information for initiating institutional change. You are part of a study of 
administrative officers in northwest community colleges who have completed a self study 
since 1990, and your attitudes and opinions are critical. 

If you have returned the questionnaire, please accept my thanks. If not, please do so 
today. We believe your response will help use better understand the use of assessment 
information for initiating institutional change. 

If you did not receive a questionnaire or it was misplaced, please call me at 206-878-3710 
ext. 3503, and I will get a new one to you immediately. 



Sincerely, 



Jim Sorensen 

Vice President for Students 
Highline Community College 
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Dear Colleague: 

About three weeks ago, I wrote to you asking your opinion on the issue of assessment and 
use of assessment information for initiating institutional change. As of today, I have not 
received your completed survey. I realize you may not have had time to complete the 
survey. I would, however, genuinely appreciate hearing from you today. 

The study is being conducted on community colleges accredited by the Northwest 
Association of Schools and Colleges who have completed a self study since 1990. Your 
response is needed to help better understand the application of assessment information in 
community colleges and the use of assessment information for initiating change. 

If you lost your questionnaire, a replacement is enclosed. I would be happy to answer any 
questions you might have about the questionnaire by calling me at 206-878-3710 ext. 
3503, or write to me at the enclosed address. 



Sincerely, 



Jim Sorensen 

Vice President for Students 
Highline Community College 
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Survey on Assessment and Institutional Change 
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CODE 



A Survey on 



ancUnstitu- 
tional Change 



Northwest Community Colleges 

Research co-sponsored by 
Highline Community College 

Please return your completed questionnaire 
in the enclosed stamped envelope to: 



Jim Sorensen, Vice President for Students MS 6-1 1 
Highline Community College 
PO Box 98000 

Des Moines, Washington 98198-9800 
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Dear Colleague, 

1 am asking for your help. 1 assume that you are as interested as 1 am at improving your institu- 
tion for students and public we serve. As part of my doctoral studies in the Community College 
Leadership Program at Oregon State University, 1 am conducting a survey about the application 
of effectiveness indicators to initiate change in Northwest Community Colleges. 1 need your 
assistance. 

The use of assessment indicators of effectiveness is an issue which has been discussed for the 
past ten years in higher education, as a way to assess our institutions and initiate change. 1 am 
interested in your opinion about the use, and benefits of using effectiveness indicators for 
initiating this change. 

You are part of a study of Presidents, Chief Instructional, Chief Student Services and Chief 
Business Affairs Administrators in Northwest Community Colleges who have completed a self 
study since 1990. Your response to this questionnaire is critical because of the small numbers of 
persons in positions such as yours. Your responses are confidential. I have coded each question- 
naire to assist in tracking the return of the questionnaire and the prompting of participants who 
have not returned their questionnaire. Your name will never be connected to your individual 
answers and will always be confidential. 

1 have enclosed a self-addressed, stamped envelope for your convenience. Please fold the 
questionnaire and return it in this envelope as soon as possible. 

Your immediate and reflective thoughts will add to the conversation on the use of assessment 
information in community colleges. 

Thank you! 



Jim Sorensen 

Vice President for Students 
Highline Community College 
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The following are categories of educational effectiveness indicators. 
Please answer sections A, B, and C as they relate to each indicator. 
Place an “X” in the appropriate location for your answer. 


A 

Perception of Indicator: 
Do you believe this 
indicator is important or 
not important as a way 
to intitiate change at 
your institution? 


Category 1 


Yes, is 
important 


No, not 
important 


1. Student Goal Attainment 

la. Measure: Number of students who leave and report their goal for 
attending college was met. 






2. Student Persistence (Fall to Fall) 

2a. Measure: Fall Quarter cohort who are still enrolled the following 
Fall Quarter and have not graduated. 






2b. Measure: Retention rates of students from quarter to quarter 
(credit students only). 






3. Program Completion 

3a. Measure: Degree and certificate completion rates. 






3b. Measure: Entering degree seeking students who complete a 
degree over time. Information is updated annually. 






Category 2 






4. Number and Rate Who Transfer 

4a. Measure: Number of entering students by cohort in a degree 
program, completing at least twelve credits, who enroll within two 
years at a four year college in a degree program. 






5. Performance After Transfer 

5a. Measure: Number of community college students transferring 
compared to native students at four year institutions who complete 
courses with a “C” or better. 






6. Success of Remedial or Developmental Students in subsequent, 
related college level course work 
6a. Measure: Proportion of entering students deficient in one or more 
basic skills (reading, writing and computation) who complete 
developmental work within one year and who complete their first 
college level course with a “C” or better. 







Please continue on the next page. 
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B 

Use of Indicator: Have you, 
yourself, used this indicator to 
Initiate change at your Institution? 

If you answer “yea”, please 
answer the corresponding 



If you have used the indicator to initiate change as you 
indicated in”B,” has the use of this indicator led to changes 
at your institution? 



question in column C. 


Don’t 

Know 


Very 

Negative 

Change 

-2 


Negative 

Change 

-1 


No 

Change 

0 


Positive 

Change 

+1 


Very 

Positive 

Change 

+2 


Yes 


No 


No, but 
plan to 
use 
































































































fffigllt 
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The following are categories of educational effectiveness indicators. 
Please answer sections A, B, and C as they relate to each indicator 
Place an "X" In the appropriate location for your answer 


A 

Perception of Indicator: 
Do you believe this 
indicator is important or 
not important as a way 
to intitiate change at 
your institution? 


Category 2 continued 


Yes, is 
important 


No, not 
important 


6b. Measure: Number of students who progress from GED, ESL and 
developmental courses to success in college level courses. 






Category 3 






7. Demonstration of General Education Outcomes 

7a. Literacy Skills 

Measure: Determined locally by your community college. 






7b. Exposure to and knowledge of the natural world (Science) 
Measure: Determined locally by your community college 






7c. Exposure to and knowledge of the arts and humanities (Humanities) 
Measure: Determined locally by your community college. 






7d. Exposure to and knowledge of culture (Social Sciences) 
Measure: Determined locally by your community college. 






7e. Cultural awareness (Pluralism/Diversity) 

Measure: Determined locally by your community college. 






7f. Computational Skills 

Measure: Determined locally by your community college. 






7g. Critical thinking and research skills 

Measure: Determined locally by your community college. 






7h. Application of skills, such as learning into action, community 
service or service learning. 

Measure: Determined locally by your community college. 






Category 4 






8. Placement Rate in the Work Force 

8a. Measure: Proportion of entering student cohort who achieve a 
"marketable skiir by completing three or more courses in an 
occupational program and who obtain employment in a field 
directly related to that skill within one year after leaving. 







Please continue on the next page. 
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B 

Use of Indicator: Have you, 
yourself, used this Indicator to 
initiate change at your institution? 

If you answer “yes”, please 



If you have used the indicator to initiate change as you 
indicated in “B t " has the use of this indicator led to 
changes at your institution? 



question in column C. 


Don’t 

Know 


Very 

Negative 

Change 

-2 


Negative 

Change 

-1 


No 

Change 

0 


Positive 

Change 

+1 


Very 

Positive 

Change 

+2 


Yes 


No 


No, but 
plan to 
use 
























cC: : ; dl: 








Ill'll *M/. 


























































/ 




























































































llllSIS 


IlliSl 








JSIISSHi 




llBBBii 


4| 3: || 





















Please turn page and continue. 



- m 



o 

ERIC 



175 



The following are categories of educational effectiveness Indicators. 
Please answer sections A t B, and C as they relate to each indicator. 
Place an “X M in the appropriate location for your answer. 


A 

Perception of Indicator: 
Do you believe this 
indicator is important or 
not important as a way 
to intitiate change at 
your institution? 


Category 4 continued 


Yes, is 
important 


No, not 
important 


9. Employer Assessment of Students 

9a. Measure: Proportion of sampled regional employers in a given field 
who indicate students from your community college have skills 
equal to or better than other employees, 






10. Client/Customer Assessment of Programs and Services 
10a. Measure: Survey, focus groups, or interviews to determine 
satisfaction of your customers, such as business and industry, 
public or private customers, about your programs, activities or 
services. 






Category 5 






1 1 . Responsiveness to Community 

11a. Measure: Survey, focus groups or interviews to identify commu- 
nity needs, your responsiveness to those needs, and a demon- 
stration that the college has met or satisfied those needs. 






12. Participation Rate in Service Level 

12a. Measure: Proportion of local population in your service area 
who have participated in an organized college activity during the 
past year. 






Category 6 






13. Student on Customer Satisfaction 

13a. Measure: Survey student satisfaction with preparation for their 
current employment one year after completing their program. 






13b. Measure: Employer satisfaction with your former students as 
graduates. 






13c. Measure: University and four year college satisfaction with 
academic preparation of your transfer students. 






13d. Measure: Student satisfaction with teaching effectiveness. 






13e. Measure: Student satisfaction with Student Services 
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B 

Use of Indicator: Have you, 
yourself, used this indicator to 
initiate change at your institution? 

If you answer “yes”, please 
answer the corresponding 



If you have used the indicator to initiate change as you 
indicated in”B,” has the use of this indicator led to changes 
at your institution? 



question in column C. 


Don’t 

Know 


Very 

Negative 

Change 

-2 


Negative 

Change 

-1 


No 

Change 

0 


Positive 

Change 

+1 


Very 

Positive 

Change 

+2 


Yes 


No 


No, but 
plan to 
use 
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ASSESSMENT 
General Questions 

14. Have you been involved in completing a full accreditation study in a community college 
accredited by the Northwest Association of Schools and Colleges, Commission on 
Colleges since 1990? 

□ Yes □ No 

1 5. Are you currently a/an 

1. □ Chief Executive Officer 

2. □ Chief Instructional Administrator 

3. □ Chief Student Services Administrator 

4. □ Chief Academic & Student Services Officer 

5. □ Chief Business Affairs/Administrative Services Officers 

6. □ Other 

15a. Do you have budget responsibility and authority for this operational area commonly 
associated with your answer to question #15 above? 

□ Yes □ No 



16. Have you in your area of responsibility used assessment information as input to the 
decision making process in any of the following areas. Check each response in which 
assessment information has been used. 



1. □ Planning 7. 

2. □ Budgeting 8. 

3. □ Program review 9. 

4. □ Accreditation 10. 

5. □ Quality improvement measures 11. 

6. □ Improving services 12. 



□ Improving learning 

□ Improving occupational programs 

□ Improving general education outcomes 

□ Legislative accountability 

□ Community accountability 

□ Other 



16a. List in priority order the number of the top five decision making processes where 
you, yourself, have considered and used assessment information for making 
decisions. 



Most important Least important 

1. 2. 3. 4. 5. 



Please continue on the next page. 
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17. Who, if any, was involved in the identification, development, and assessment of effec- 
tiveness indicators at your institutions (Mark one) 



1. □ Faculty 

2. □ Administration 

3. □ Classified Staff 

4. □ Students 

5. □ All the Above 

6. □ Don’t Know 

7. □ Other Combination 

18. Please identify positive changes or improvements which may have occurred at your 
institution by using effectiveness indicators. 

Positive Changes: 



Negative Changes: 



19. How large is your institution in total FTEs? (Include all fund sources) 

1. □ 1-2,499 

2. □ 2,500-4,999 

3. □ 5,000-7,499 

4. □ 7,500-9,999 

5 . □ 10 , 000 + 

20. Please describe your college as mostly serving an urban or rural population? 

1. □ Mostly or all Rural 

2. □ Suburban 

3. □ Mostly or all Urban 

21 . Please describe any concerns about the collection and use of institutional effectiveness 
information at your institution for initiating institutional change? 



Please turn page and continue . 





22. Please describe any concerns about the use of institutional effectiveness information by 
your publics, such as the local community, legislators, and public or private agencies? 



23. Do you wish to receive a copy of the results? □ Yes □ No 

24. Comments: 



Thank you for responding and returning this questionaire. 
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